Keyboard Shortcuts
Likes
- Wallaroo
- Messages
Search
How to read and write MQTT msgs from a Wallaroo python app?
Hi all, We are currently trying to evaluate Wallaroo to see if it is suitable for our streaming framework. We want to write a python streaming Wallaroo app that needs to read messages from MQTT topics, do some processing and write out to other MQTT topics. However it is not very clear what is the best/easiest way to do that. It seems Wallaroo is using a slightly more involved way of streaming input and output than other frameworks like Flink, Spark, Storm where you have standard interfaces for sources and sinks which can be implemented to read and write from any systems using high level clients for those particular systems/protocols. With Wallaroo the process seems a lot more low level. From what we understand we need to write a python decoder and encoder for lets say raw MQTT msg bytes for example. But we would still need to write another piece of code/program that uses an MQTT client (in some language ... python or pony?) to subscribe to topics and send them on the TCP port where the app source is listening. Is that correct? Is there a python interface for that? Or does that need to be a Pony component? What is the lifecycle of that component? Could not find a tutorial in the docs that shows how to add a support for a new protocol start to finish, is there something like that? Sorry for the long message, thanks! M |
Re: How to read and write MQTT msgs from a Wallaroo python app?
Sean T. Allen
Hi Martin, Thanks for your questions. Didn't feel like a long message at all. We have a project that we call "Bring Your Own Integrations" that is currently underway. A preview version should be released in two to three weeks. Bring Your Own Integrations, or BYOI as we call it, will allow Wallaroo users like you to write source and sinks in Python (or any other language). The results of BYOI are definitely what you are looking for. Currently, you would need to use the TCP support to write bridge applications that understand how to read and write MQTT and then send/receive from Wallaroo over TCP. You can see examples of this (not MQTT) in our streaming Twitter application example where there is a Python "client" that reads from the Twitter API and writes to Wallaroo over TCP. The twitter example code is here: The specific "client" that I'm referencing is here: When BYOI is released, there will be full documentation on how to use. We'd love to work with you on getting MQTT working and help you evaluate Wallaroo. We'd be happy to do a call or more private emails to discuss your use-case and help you make it a success. When the preview release of BYOI is released, we'll be announcing it here on the mailing list as well as elsewhere. -Sean- |
Need your review and guide : DDoS Detection using Wallaroo
Hi. I have a problem in setup the wallaroo environment for ubuntu trusty. I am trying to run the ddos detection using wallaroo but it fails . I got some error when enter the make command in machida directory. For ubuntu in virtual box , some error like /bin/sh : stable not found? make***? ?(like that) and for ubuntu in vmware I got error like? cat VERSION .. No LLVM installation found. Stop Could you help me solve this problem. TQ |
Re: Need your review and guide : DDoS Detection using Wallaroo
Sean T. Allen
Hi Farzana, Sorry to hear you are having trouble, What directions did you use to setup your virtual box Wallaroo environment? Did you use these directions? If not, what directions did you use? -Sean- On Sun, Jul 22, 2018 at 2:02 PM farzana zakaria <farzanazakaria96@...> wrote:
|
Re: Need your review and guide : DDoS Detection using Wallaroo
Hi, I'm using this direction??on virtualbox and vmware workstation? I think I having trouble with pony compiler . It is necessary to install ponyc and pony-stable. I don know whether I install the library in wrong directory or what . I just follow the instruction given.? When I run make command in machida , its gonna be a few error On Mon, Jul 23, 2018 at 6:40 AM, Sean T. Allen <sean@...> wrote:
|
Re: Need your review and guide : DDoS Detection using Wallaroo
Sean T. Allen
You are definitely missing some things that the directions instruct you to install. Id suggest either starting over with a new image and make sure everything in the directions works without error. The other option is to use the Vagrantfile that we provide and for which, there are instructions to use. Those instructions are in my first email. On Sun, Jul 22, 2018, 19:50 farzana zakaria <farzanazakaria96@...> wrote:
|
Re: Need your review and guide : DDoS Detection using Wallaroo
Hi,? Thank you for your respond. If I want to start over using vagrant file , I have to use Ubuntu Xenial(16.04) right ? And which platform i can use ? Whether ubuntu in virtualbox on or vmware platform or both i can use ? On Mon, Jul 23, 2018 at 11:07 AM, Sean T. Allen <sean@...> wrote:
|
Re: Need your review and guide : DDoS Detection using Wallaroo
Sean T. Allen
We test the supplied Vagrantfile using virtual box. In theory, VMWare should work as well. I'm away from my computer at the moment, but, yes, I believe the Vagrantfile file used Xenial. On Sun, Jul 22, 2018, 20:14 farzana zakaria <farzanazakaria96@...> wrote:
|
Re: Need your review and guide : DDoS Detection using Wallaroo
Thank you for your respond. For setting up environment wallaroo in vagrant, according to the instruction when clone the wallaroo and enter the wallaroo directory, I have to install vagrant and install the package. Then up the vagrant . Do I have to install again the ponyc , libsnappy, python development library and etc .. in order to run the ddos detection ? Thank you for your consideration On Mon, Jul 23, 2018 at 11:37 AM, Sean T. Allen <sean@...> wrote:
|
Re: Need your review and guide : DDoS Detection using Wallaroo
Hi. I'm starting over with a new image and follow the direction as well . But the error keep appear . The installation for ponyc in the direction seem can be apply in debian linux not ubuntu linux . I installed ponyc using source and it seems work . Could you test the example ddos detection using wallaroo and run it again .?
toggle quoted message
Show quoted text
On Monday, July 23, 2018, Sean T. Allen <sean@...> wrote:
|
Re: Need your review and guide : DDoS Detection using Wallaroo
Sean T. Allen
The Vagrantfile that we provide installs everything you need. Note, the DDos example uses an older version on the Wallaroo API and would need to be updated. On Sun, Jul 22, 2018, 22:02 farzana zakaria <farzanazakaria96@...> wrote:
|
Re: Need your review and guide : DDoS Detection using Wallaroo
Thank you for your respond. Means that I dont have to install the other dependencies for wallaroo ? How wallaroo API gonna be update ? Can I apply ddos example for the older version or not ? On Mon, Jul 23, 2018 at 9:44 PM, Sean T. Allen <sean@...> wrote:
|
Re: Need your review and guide : DDoS Detection using Wallaroo
Sean Allen
The Vagrant directions will create a Vagrant VM with the latest version of Wallaroo installed. The DDos example README has notes on requirements: The important bit from that is under requirements: "?NOTE:?This is incompatible with Wallaroo 0.4.0 and up due to an API change, see??to bring this example up to date if needed." In particular, this means that the Wallaroo framework API has had breaking changes since that example was published and it hasn't been updated to the latest API. I hope this answers your questions. -Sean- On Mon, Jul 23, 2018 at 7:17 AM farzana zakaria <farzanazakaria96@...> wrote:
|
Re: Need your review and guide : DDoS Detection using Wallaroo
Thank you for the answer. Could you please assist me to been able to apply ddos detection example for wallaroo please.? On Mon, Jul 23, 2018 at 10:38 PM, Sean Allen <sean@...> wrote:
|
Re: Need your review and guide : DDoS Detection using Wallaroo
Sean Allen
What is it you are looking to do Farzana? Is there a reason you are particularly interested in the DDos example rather than one of the more recent examples? -Sean- On Mon, Jul 23, 2018 at 7:46 AM farzana zakaria <farzanazakaria96@...> wrote:
|
Need your review and guide : DDoS Detection using Wallaroo
According to? , after installation the libsnappy in wallaroo directory , it change it directory to lz4-1.7.5( outside the wallaroo) . It is the python library and the docker have to be install in that lz4-1.7.5 directory ? ?
toggle quoted message
Show quoted text
On Monday, July 23, 2018, Sean Allen <sean@...> wrote:
|
Re: Need your review and guide : DDoS Detection using Wallaroo
On Mon, Jul 23, 2018 at 12:12 PM, farzana zakaria <farzanazakaria96@...> wrote: According to? , after installation the libsnappy in wallaroo directory , it change it directory to lz4-1.7.5( outside the wallaroo) . It is the python library and the docker have to be install in that lz4-1.7.5 directory ?? Hi, Farzana.? The liblz4 instructions for Ubuntu Trusty/14.x include a final `sudo make install` step.? The `make install` step will copy all necessary liblz4 files to the Ubuntu system's default directories.? You can delete the ~/lz4-1.7.5 directory afterward. -Scott |
Re: Need your review and guide : DDoS Detection using Wallaroo
Hi. Thank you for the answer. It is means that the installation of python library and the docker is then have to be install in /wallaroo-tutorial/wallaroo/ directory ??
toggle quoted message
Show quoted text
On Tuesday, July 24, 2018, Scott Lystig Fritchie <scott@...> wrote:
|
Re: Need your review and guide : DDoS Detection using Wallaroo
Sean T. Allen
No. Those commands will install it elsewhere. On Mon, Jul 23, 2018, 16:14 farzana zakaria <farzanazakaria96@...> wrote: Hi. Thank you for the answer. It is means that the installation of python library and the docker is then have to be install in /wallaroo-tutorial/wallaroo/ directory ?? |