wallaroo@groups.io

How to read and write MQTT msgs from a Wallaroo python app?

Hi all,

We are currently trying to evaluate Wallaroo to see if it is suitable for our streaming framework.

We want to write a python streaming Wallaroo app that needs to read messages from MQTT topics, do some processing and write out to other MQTT topics.

However it is not very clear what is the best/easiest way to do that.

It seems Wallaroo is using a slightly more involved way of streaming input and output than other frameworks like Flink, Spark, Storm where you have standard interfaces for sources and sinks which can be implemented to read and write from any systems using high level clients for those particular systems/protocols.

With Wallaroo the process seems a lot more low level. From what we understand we need to write a python decoder and encoder for lets say raw MQTT msg bytes for example. But we would still need to write another piece of code/program that uses an MQTT client (in some language ... python or pony?) to subscribe to topics and send them on the TCP port where the app source is listening. Is that correct? Is there a python interface for that? Or does that need to be a Pony component? What is the lifecycle of that component? Could not find a tutorial in the docs that shows how to add a support for a new protocol start to finish, is there something like that?

Sorry for the long message, thanks!

M

Re: How to read and write MQTT msgs from a Wallaroo python app?

Sean T. Allen

#42

Hi Martin,

Thanks for your questions. Didn't feel like a long message at all.

We have a project that we call "Bring Your Own Integrations" that is currently underway. A preview version should be released in two to three weeks. Bring Your Own Integrations, or BYOI as we call it, will allow Wallaroo users like you to write source and sinks in Python (or any other language). The results of BYOI are definitely what you are looking for. Currently, you would need to use the TCP support to write bridge applications that understand how to read and write MQTT and then send/receive from Wallaroo over TCP. You can see examples of this (not MQTT) in our streaming Twitter application example where there is a Python "client" that reads from the Twitter API and writes to Wallaroo over TCP.

The twitter example code is here:

The specific "client" that I'm referencing is here:

When BYOI is released, there will be full documentation on how to use. We'd love to work with you on getting MQTT working and help you evaluate Wallaroo. We'd be happy to do a call or more private emails to discuss your use-case and help you make it a success.

When the preview release of BYOI is released, we'll be announcing it here on the mailing list as well as elsewhere.

-Sean-

July 12 blog post: Erlang, Wallaroo, and Spambots

#43

Good morning! There's a new blog post on our engineering blog at Wallaroo Labs, in which we connect an XMPP chat server to Wallaroo, and fight the good fight against spam. Check it out:

Need your review and guide : DDoS Detection using Wallaroo

#44

Hi. I have a problem in setup the wallaroo environment for ubuntu trusty. I am trying to run the ddos detection using wallaroo but it fails . I got some error when enter the make command in machida directory. For ubuntu in virtual box , some error like /bin/sh : stable not found?

make***? ?(like that)

and for ubuntu in vmware I got error like?

cat VERSION ..

No LLVM installation found. Stop

Could you help me solve this problem. TQ

Re: Need your review and guide : DDoS Detection using Wallaroo

Sean T. Allen

#45

Hi Farzana,

Sorry to hear you are having trouble,

What directions did you use to setup your virtual box Wallaroo environment?

Did you use these directions?

If not, what directions did you use?

-Sean-

开云体育