Re: [wallaroo - Goland] - Go API documentation and samples
HI Sean?
Thanks for the information .
On Wed, Apr 3, 2019 at 6:57 AM Sean T. Allen < sean@...> wrote: It's something that we would discuss. If it all passed with the latest master, we'd probably merge it but, I couldn't promise ongoing support. We stopped developing the Go API so we could move quicker with adding features in Python (and Pony) where there has either been demand from clients or from ourselves to support clients. If you are interested in supporting the Go API, we should have a longer conversation where we could discuss what that would look like and what your expectations would be.
If you'd like to have that conversation, you can email me on my work email at sean@.... Yes, I would like to have that conversation, will email to you?
/Jay? On Tue, Apr 2, 2019 at 9:00 PM Jasintha Dassanayake < dassanayake@...> wrote: Hi devs
If we migrate Go APIs into the latest version (0.6.0) and send a PR, will you guys accept and merge it??
Thanks Jay
On Wed, Apr 3, 2019 at 6:01 AM Jasintha Dassanayake via Groups.Io <dassanayake= [email protected]> wrote: Hi Sean?
Thanks for the quick reply, then we will be using pony or python, BTW, is there any plan to support RUST in the?future?
thanks /Jay
On Wed, Apr 3, 2019 at 5:40 AM Sean Allen < sean@...> wrote: Hi Jay,
We aren't currently supporting the Go API. There wasn't much demand for it. We'd be willing to entertain support on a commercial basis. For the open source, version, we have Python support as there was a lot more demand. The last update to the Go API was for version 0.5.4.
There was a large API update in 0.6.0 that we didn't think warranted investing in changing the Go API.
If you check out the 0.5.4 Wallaroo tag, all the documentation would be available. You'd then be able to use Gitbook to generate a web version of the documentation or you could work with the markdown files in the `book` directory directly.
-Sean-
--
Jasintha Dassanayake , Co-founder and Principal Engineer at xiges? T:? +94 115 92 6090
? ?  ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Delivery excellence guaranteed
--
Jasintha Dassanayake , Co-founder and Principal Engineer at xiges? T:? +94 115 92 6090
? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Delivery excellence guaranteed
-- Jasintha Dassanayake , Co-founder and Principal Engineer at xiges? T:? +94 115 92 6090
? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Delivery excellence guaranteed
|
Re: [wallaroo - Goland] - Go API documentation and samples
It's something that we would discuss. If it all passed with the latest master, we'd probably merge it but, I couldn't promise ongoing support. We stopped developing the Go API so we could move quicker with adding features in Python (and Pony) where there has either been demand from clients or from ourselves to support clients. If you are interested in supporting the Go API, we should have a longer conversation where we could discuss what that would look like and what your expectations would be.
If you'd like to have that conversation, you can email me on my work email at sean@....
On Tue, Apr 2, 2019 at 9:00 PM Jasintha Dassanayake < dassanayake@...> wrote: Hi devs
If we migrate Go APIs into the latest version (0.6.0) and send a PR, will you guys accept and merge it??
Thanks Jay
On Wed, Apr 3, 2019 at 6:01 AM Jasintha Dassanayake via Groups.Io <dassanayake= [email protected]> wrote: Hi Sean?
Thanks for the quick reply, then we will be using pony or python, BTW, is there any plan to support RUST in the?future?
thanks /Jay
On Wed, Apr 3, 2019 at 5:40 AM Sean Allen < sean@...> wrote: Hi Jay,
We aren't currently supporting the Go API. There wasn't much demand for it. We'd be willing to entertain support on a commercial basis. For the open source, version, we have Python support as there was a lot more demand. The last update to the Go API was for version 0.5.4.
There was a large API update in 0.6.0 that we didn't think warranted investing in changing the Go API.
If you check out the 0.5.4 Wallaroo tag, all the documentation would be available. You'd then be able to use Gitbook to generate a web version of the documentation or you could work with the markdown files in the `book` directory directly.
-Sean-
--
Jasintha Dassanayake , Co-founder and Principal Engineer at xiges? T:? +94 115 92 6090
? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Delivery excellence guaranteed
--
Jasintha Dassanayake , Co-founder and Principal Engineer at xiges? T:? +94 115 92 6090
? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Delivery excellence guaranteed
|
Re: [wallaroo - Goland] - Go API documentation and samples
We haven't had any Rust conversations. It isn't something that anyone has requested. If someone in a commercial relationship requested it, it would be something we considered.
On Tue, Apr 2, 2019 at 8:31 PM Jasintha Dassanayake < dassanayake@...> wrote: Hi Sean?
Thanks for the quick reply, then we will be using pony or python, BTW, is there any plan to support RUST in the?future?
thanks /Jay
On Wed, Apr 3, 2019 at 5:40 AM Sean Allen < sean@...> wrote: Hi Jay,
We aren't currently supporting the Go API. There wasn't much demand for it. We'd be willing to entertain support on a commercial basis. For the open source, version, we have Python support as there was a lot more demand. The last update to the Go API was for version 0.5.4.
There was a large API update in 0.6.0 that we didn't think warranted investing in changing the Go API.
If you check out the 0.5.4 Wallaroo tag, all the documentation would be available. You'd then be able to use Gitbook to generate a web version of the documentation or you could work with the markdown files in the `book` directory directly.
-Sean-
--
Jasintha Dassanayake , Co-founder and Principal Engineer at xiges? T:? +94 115 92 6090
? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Delivery excellence guaranteed
|
Re: [wallaroo - Goland] - Go API documentation and samples
Hi devs
If we migrate Go APIs into the latest version (0.6.0) and send a PR, will you guys accept and merge it??
Thanks Jay
On Wed, Apr 3, 2019 at 6:01 AM Jasintha Dassanayake via Groups.Io <dassanayake= [email protected]> wrote: Hi Sean?
Thanks for the quick reply, then we will be using pony or python, BTW, is there any plan to support RUST in the?future?
thanks /Jay
On Wed, Apr 3, 2019 at 5:40 AM Sean Allen < sean@...> wrote: Hi Jay,
We aren't currently supporting the Go API. There wasn't much demand for it. We'd be willing to entertain support on a commercial basis. For the open source, version, we have Python support as there was a lot more demand. The last update to the Go API was for version 0.5.4.
There was a large API update in 0.6.0 that we didn't think warranted investing in changing the Go API.
If you check out the 0.5.4 Wallaroo tag, all the documentation would be available. You'd then be able to use Gitbook to generate a web version of the documentation or you could work with the markdown files in the `book` directory directly.
-Sean-
--
Jasintha Dassanayake , Co-founder and Principal Engineer at xiges? T:? +94 115 92 6090
? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Delivery excellence guaranteed
-- Jasintha Dassanayake , Co-founder and Principal Engineer at xiges? T:? +94 115 92 6090
? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Delivery excellence guaranteed
|
Re: [wallaroo - Goland] - Go API documentation and samples
Hi Sean?
Thanks for the quick reply, then we will be using pony or python, BTW, is there any plan to support RUST in the?future?
thanks /Jay
On Wed, Apr 3, 2019 at 5:40 AM Sean Allen < sean@...> wrote: Hi Jay,
We aren't currently supporting the Go API. There wasn't much demand for it. We'd be willing to entertain support on a commercial basis. For the open source, version, we have Python support as there was a lot more demand. The last update to the Go API was for version 0.5.4.
There was a large API update in 0.6.0 that we didn't think warranted investing in changing the Go API.
If you check out the 0.5.4 Wallaroo tag, all the documentation would be available. You'd then be able to use Gitbook to generate a web version of the documentation or you could work with the markdown files in the `book` directory directly.
-Sean-
-- Jasintha Dassanayake , Co-founder and Principal Engineer at xiges? T:? +94 115 92 6090
? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Delivery excellence guaranteed
|
Re: [wallaroo - Goland] - Go API documentation and samples
Hi Jay,
We aren't currently supporting the Go API. There wasn't much demand for it. We'd be willing to entertain support on a commercial basis. For the open source, version, we have Python support as there was a lot more demand. The last update to the Go API was for version 0.5.4.
There was a large API update in 0.6.0 that we didn't think warranted investing in changing the Go API.
If you check out the 0.5.4 Wallaroo tag, all the documentation would be available. You'd then be able to use Gitbook to generate a web version of the documentation or you could work with the markdown files in the `book` directory directly.
-Sean-
|
[wallaroo - Goland] - Go API documentation and samples
Hi Devs? We are a startup and working on a software platform. We have decided to use wallaroo?as a stream processor in our platform.
We would like to try out the Go API instead of using python or pony API. However, I only found this?? So, are you discontinuing the go support? Else could you point me Go api documentation?
Thanks /Jay -- Jay Co-founder and CTO at xiges T:? +94 115 92 6090
? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Delivery excellence guaranteed
|
The connector protocol is language agnostic. You can write a connector in any language you want, not just Python.
For a specific connector, you could pickle the data and send it across as long as you knew that the receiver can deserialize it.
The idea that the Python library support for connectors should default to pickling for serialization is an interesting one, in the context of the Python library support.
Seems like a good issue for the Github issue tracker where it could be discussed more fully.
Thanks for the prompt reply
Why not default to pickling for connector encode/decode ? as communication between worker already default to pickling.
|
Thanks for the prompt reply
Why not default to pickling for connector encode/decode ? as communication between worker already default to pickling.
|
Hi
Just wanting to ask, what is the benefit of writing connector in pony rather than python ?
So I manage writing connector for nats server (after digging through the source code), then I found that the connector is a separate process, if it is in pony, it will be in one process right ?
Connectors are all external processes. You could write a NATs source in Pony that runs directly in the Wallaroo binary. For that, you'd need to write Pony code that gets compiled directly into Wallaroo. You'd probably also have to change the initialization process and a variety of other things. In the end, not something most people are probably going to want to do.?
Connectors are separate processes which allows the writer to only worry about the connector, not any Wallaroo plumbing. ?
The second, the connector api is marked experimental, any plan on breaking change ?_._,_. It's possible based on feedback and usage that we will end up making breaking changes. There are planned changes around resilience, replayability, and more thoroughly building those into the protocol.
|
Hi
Just wanting to ask, what is the benefit of writing connector in pony rather than python ?
So I manage writing connector for nats server (after digging through the source code), then I found that the connector is a separate process, if it is in pony, it will be in one process right ?
The second, the connector api is marked experimental, any plan on breaking change ?
|
Re: Need Wallaroo help? Talk to me
Hi Audun
It's clear to me that what you want is a . You'd have a connector process that opens a python socket to the data stream, and exposes that same stream to Wallaroo. I imagine it's not a far cry from your GenSource modification.???is the general pattern you'll want to use, but instead of subscribing to a Redis topic, you'd just open a tcp socket to the broadcast server.
Secondly, regarding framing: the reason for the framing format is so that the decode function can be stateless. Otherwise you'd have to buffer extraneous data and decide when there's 'enough' to emit a business object. With the frame header, the decode function is only called once the amount of bytes defined in the header has been ingested off the TCP socket. You won't have to worry about this if you go ahead and use a Connector.?
If you do want to use framing, an easy trick is to do the following to your send your framed messages to the TCP Source:
socket = ... (establish the socket) line = "This data will be received in one go in the decode function" socket.sendall(struct.pack(">I",len(line))+line)
Cheers,? Simon
(P.S. Sorry about the blank email above -- still getting used to the new gmail interface)
On Fri, Jan 18, 2019 at 4:37 PM Simon Zelazny < simon@...> wrote:
Yes, I know that's the intended use of GenSourceConfig. I just tweaked the mode of generation to be that of
popping elemtns off a TCP stream.
?
I'll provide you the details later if necessary, but in order not to get sidetracked let me remind you of my? original?
general question: how does one? redirect an external TCP stream to wallaroo?
I read up on framed messages and the Giles script. But Giles seems to only work for files.
What I have is info continuously coming across a TCP socket and I want to process it using wallaroo.? Do I need to frame?
each individual element manually?
Audun
|
Re: Need Wallaroo help? Talk to me
Yes, I know that's the intended use of GenSourceConfig. I just tweaked the mode of generation to be that of
popping elemtns off a TCP stream.
?
I'll provide you the details later if necessary, but in order not to get sidetracked let me remind you of my? original?
general question: how does one? redirect an external TCP stream to wallaroo?
I read up on framed messages and the Giles script. But Giles seems to only work for files.
What I have is info continuously coming across a TCP socket and I want to process it using wallaroo.? Do I need to frame?
each individual element manually?
Audun
|
Re: Need Wallaroo help? Talk to me
Yes, I know that's the intended use of GenSourceConfig. I just tweaked the mode of generation to be that of
popping elemtns off a TCP stream.
?
I'll provide you the details later if necessary, but in order not to get sidetracked let me remind you of my? original?
general question: how does one? redirect an external TCP stream to wallaroo?
I read up on framed messages and the Giles script. But Giles seems to only work for files.
What I have is info continuously coming across a TCP socket and I want to process it using wallaroo.? Do I need to frame?
each individual element manually?
Audun
|
Re: Need Wallaroo help? Talk to me
In the meantime I managed to hook up the stream by using?
wallaroo.GenSourceConfig and then just popping one object off the stream at a time as a way of generating input.
Can you explain a bit further what you mean by this?? What are you passing into?wallaroo.GenSourceConfig? wallaroo.GenSourceConfig is meant to accept a class which autogenerates messages for Wallaroo.?
If possible, providing us with a Github Gist of your application might make it easier for us to help you out. Sorry mistyped, the port was 7010 in both instances.
Thanks for the link I'll read up.
In the meantime I managed to hook up the stream by using?
wallaroo.GenSourceConfig and then just popping one object off the stream at a time as a way of generating input.
Any reason why I shouldn't do it that way?
|
Re: Need Wallaroo help? Talk to me
Sorry mistyped, the port was 7010 in both instances.
Thanks for the link I'll read up.
In the meantime I managed to hook up the stream by using?
wallaroo.GenSourceConfig and then just popping one object off the stream at a time as a way of generating input.
Any reason why I shouldn't do it that way?
|
Re: Need Wallaroo help? Talk to me
I started the Wallaroo module:?
#!/usr/bin/env python
import os
? ?
instruction=("machida "
? ? ? ? ? ? ?"--application-module tester "
? ? ? ? ? ? ?"--in "?
? ? ? ? ? ? ?"--out "
? ? ? ? ? ? ?"--metrics "
? ? ? ? ? ? ?"--control "
? ? ? ? ? ? ?"--data "
? ? ? ? ? ? ?"--name worker-name "
? ? ? ? ? ? ?"--external "
? ? ? ? ? ? ?"--cluster-initializer "
? ? ? ? ? ? ?"--ponythreads=1 "
? ? ? ? ? ? ?"--ponynoblock"
? ? ? ? ? ? ?)
os.system(instruction)
Using the a module that throughputs input to putput uncanged. The I listen to 7002 with?
#!/usr/bin/env python
import os
?
instruction = ("data_receiver "
? ? ? ? ? ? ? ?"--ponythreads=1 "
? ? ? ? ? ? ? ?"--ponynoblock "?
? ? ? ? ? ? ? ?"--listen "
? ? ? ? ? ? ? ?)
?
os.system(instruction)
At this point, wallaroo reports the following:
>> |~~ INIT PHASE I: Application is created! ~~|
>> |~~ INIT PHASE II: Application is initialized! ~~|
>> TCPSink initializing connection to
>> TCPSink connected
>> |~~ INIT PHASE III: Application is ready to work! ~~|
>> Inputter source attempting to listen on
>> Inputter source is listening
>> initializer reported topology ready!
>> All 1 workers reporting Topology ready!
>> Application has successfully initialized.
?
Now, this means wallaroo is listening on 7010 and that 7010 is correctly configured as a TCP input source, right? I can now connect with netcat? >> nc 127.0.0.1 7010 ... and wallaroo confirms connection:
>> Inputter source: accepted a connection
Yet, nothing I send over the connection is throughput by the wallaroo module to 7002. For instance I can redirect input from a file like this:
>> nc 127.0.0.1 8000 < text.txt
Is that the nc command you are using to send something into Wallaroo? If yes, that wouldnt work as its running on port 8000.
If you are sending via nc to port 7010, then Wallaroo is receiving the data. The first question would be, are you sending a framed message? Probably not if using nc.?
The TcpSource accepts framed messages where there is:
4 byte header that indicates the length of the payload Payload
If you are interested in sending in a file, line by line, I'd suggest checking out the Giles Sender tool that comes with Wallaroo:
It can read a file and send it line by line adding the appropriate framing. ?
|
Re: Need Wallaroo help? Talk to me
I started the Wallaroo module:?
#!/usr/bin/env python
import os
? ?
instruction=("machida "
? ? ? ? ? ? ?"--application-module tester "
? ? ? ? ? ? ?"--in 127.0.0.1:7010 "?
? ? ? ? ? ? ?"--out 127.0.0.1:7002 "
? ? ? ? ? ? ?"--metrics 127.0.0.1:5001 "
? ? ? ? ? ? ?"--control 127.0.0.1:6000 "
? ? ? ? ? ? ?"--data 127.0.0.1:6001 "
? ? ? ? ? ? ?"--name worker-name "
? ? ? ? ? ? ?"--external 127.0.0.1:5050 "
? ? ? ? ? ? ?"--cluster-initializer "
? ? ? ? ? ? ?"--ponythreads=1 "
? ? ? ? ? ? ?"--ponynoblock"
? ? ? ? ? ? ?)
os.system(instruction)
Using a module that throughputs input to output unchanged. Then I listen to 7002:??
#!/usr/bin/env python
import os
?
instruction = ("data_receiver "
? ? ? ? ? ? ? ?"--ponythreads=1 "
? ? ? ? ? ? ? ?"--ponynoblock "?
? ? ? ? ? ? ? ?"--listen 127.0.0.1:7002"
? ? ? ? ? ? ? ?)
?
os.system(instruction)
At this point, wallaroo reports:
>> |~~ INIT PHASE I: Application is created! ~~|
>> |~~ INIT PHASE II: Application is initialized! ~~|
>> TCPSink initializing connection to 127.0.0.1:7002
>> TCPSink connected
>> |~~ INIT PHASE III: Application is ready to work! ~~|
>> Inputter source attempting to listen on 127.0.0.1:7010
>> Inputter source is listening
>> initializer reported topology ready!
>> All 1 workers reporting Topology ready!
>> Application has successfully initialized.
?
This means that wallaroo is listening on 7010 and that 7010 is correctly configured as a TCP input source, right? I can now connect with netcat? >> nc 127.0.0.1 7010 ... and wallaroo confirms connection:
>> Inputter source: accepted a connection
Yet, nothing I send over that connection is throughput by the wallaroo module to 7002. For instance, if I redirect input from a file like this:
>> nc 127.0.0.1 8000 < text.txt
nothing happens. Any idea why that is? After all wallaroo confirms the connection as an input source.
A
?
|
Re: Need Wallaroo help? Talk to me
Sorry, can you go into more detail what you did with `nc` including what other commands you ran and what happened?
In fact, if I just do
nc 127.0.0.1 7010 (corresponding to the adress specified as input source)
then my wallaroo module acknowledges the connection, but no data I send over it gets through.
A
-- Sean T. Allen VP of Engineering WallarooLabs.com
Unlock Data's Potential. Wallaroo Labs makes it simple to scale data applications efficiently, reliably, and on-demand - without worrying about infrastructure. Get to production fast, innovate rapidly, and operate at a low cost.
Please star us on GitHub.
|
Re: Need Wallaroo help? Talk to me
In fact, if I just do
nc 127.0.0.1 7010 (corresponding to the adress specified as input source)
then my wallaroo module acknowledges the connection, but no data I send over it gets through.
A
|
