• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

nio and file transfers

 
Ranch Hand
Posts: 2120
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am learning nio and I want to share my testing code.
FileTX transmits files to several remote hosts using blocking channels.
FileRX receives files using blocking channels as well.
FileRX3 uses classic io. I hope some examples using Selector will
follow in the next batch.
Before the code a summary of the "conclusions". Feel free to correct me
where needed. Though I think they match the API descriptions and the
behaviour of the code.
1) Configuring several SocketChannels to non blocking we can call
connect on them, wait simultaneously for such connections, and later
check them with finishConnect. This can be done with either a
Selector or polling (see FileTX.waitForConnects)
2) If socketChannel is in blocking mode
fileChannel.transferTo(filePosition, nbytes, socketChannel)
will return only after having transferred nbytes from the given
position in the file to the socket. However we cannot make
filePosition equals to zero, and nbytes equals to file length if the
file is large (110MB) because an exception is thrown. Thus we need a
loop ( see FileTX.sendToAll(FileChannel) )
3) If socketChannel is in blocking mode socketChannel.write(ByteBuffer)
will return only after having writen to the socket
ByteBuffer.remaining bytes ( see FileTX.sendToAll(ByteBuffer[]) )
4) If socketChannel is in blocking mode socketChannel.read(ByteBuffer)
will wait untill some bytes arrives to the socket, but it will return
before ByteBuffer is full if there is no more bytes in the reception
buffer of the socket. Thus we need a loop for partial readings
( see Protocol.process )
5) If socketChannel is in blocking mode
fileChannel.transferFrom(socketchannel, 0, fileLength) will return
after having writen to disk the whole file from the socket
(see Protocol.process ) Nice method!
6) nio was quicker than io. FileRX received a file of 450MB sent by
FileTX in 22 min. FileRX3 took 23 min. There was no variation by
buffering FileTX3's reading of the socket, or writing to the disk or
both.
Notice that for the sake of brevity, the examples do not follow some OO principles and patterns. Rather than that they concentrate on showing nio.
[ April 30, 2004: Message edited by: Jose Botella ]
 
Jose Botella
Ranch Hand
Posts: 2120
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
All the testing was performed in Windows machines. The programs do
not worry about file's content byte order or charsets for the "protocol".
The "protocol" I used was very simple and fragile. FileTX firstly sends
one int (BE), specifying the length of the subsequent information to
send; names of files separated by tabs from its length, and a new line
before the next name. Then the content of each file in the same order is
sent. This implies that if an IOException occurs we lost the
"synchronization" and the rest of the files to be received are lost
because we abort.
However both FileRX and FileRX3 continue reading from the socket in the
event of a FileNotFoundException at the time of file creation. The
reading is discarded and the "byte synchronization" is preserved in
order to read the following files.
 
Jose Botella
Ranch Hand
Posts: 2120
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
 
Jose Botella
Ranch Hand
Posts: 2120
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
 
Jose Botella
Ranch Hand
Posts: 2120
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Now is the turn of Selector. It follows some guidelines about its use. Then some extracts from the API, which I belief are pplaced in the correct order to be more easily understandable. And lastly an example program.

****************************************************************************

The way to go with a Selector is removing the SelectionKey from the
selected set after getting it from the iterator returned by
selectedKeys(). In order to update its interest set before calling select
again call either Selector.register or SelectionKey.interestOps
Do not call SelectionKey.cancel before registering again. Use this
method to cancel the registration definitely. A single thread should perform the IO activity related with a given Selector, and to update its interests if necessary.

Once a socketChannel is registered with a selector for write we must
delete such interest after the writing operation. Otherwise, because
a socket is almost always writable --unless there were network
congestion and the TCP send buffer is full-- we would find a loop;
because the selector keeps on notifying the availabity to write. Next
time we want to write register that interest again. Either by
SocketChannel.register or by SelectionKey.interestOps
However the read operations do not need to be deregistered after being performed because the selector only notifies when data is available in the channel.

A way to read a non-blocking channel could be:


*************************************************************************

This is summary of the API:

***********************************************************************

Selection Key

A selection key is created each time a channel is registered with a
selector. A key remains valid until it is cancelled by invoking its
cancel method, by closing its channel, or by closing its selector.
Cancelling a key does not immediately remove it from its selector;
it is instead added to the selector's cancelled-key set for removal
during the next selection operation.

A selection key contains two operation sets represented as integer
values:
a) The interest set determines which operation will be
tested for readiness the next time one of the selector's selection
methods is invoked. The interest set is initialized with the value
given when the key is created; it may later be changed via the
interestOps(int) method
b) The ready set identifies the operation categories for which the
key's channel has been detected to be ready by the key's selector.
The ready set is initialized to zero when the key is created. But it
cannot be modified directly.


That a selection key's ready set indicates that its channel is ready for
some operation category is a hint, but not a guarantee, that an
operation in such a category may be performed by a thread without
causing the thread to block. A ready set is most likely to be accurate
immediately after the completion of a selection operation. It is likely
to be made inaccurate by external events and by I/O operations that are
invoked upon the corresponding channel.

The operations of reading and writing the interest set will, in general,
be synchronized with certain operations of the selector. Exactly how is
implementation dependant: In a naive implementation, reading or writing
the interest set may block indefinitely if a selection operation is
already in progress.

cancel()
It synchronizes on the selector's cancelled-key set, and therefore may
block briefly if invoked concurrently with a cancellation or selection
operation involving the same selector.

interestOps()
Whether or not it blocks, and for how long, is implementation-dependent.

interestOps(int)
Whether or not it blocks, and for how long, is implementation-dependent.


**********************************************************************

SelectableChannel

Once registered with a selector, a channel remains registered until it
is deregistered. This involves deallocating whatever resources were
allocated to the channel by the selector.

A channel may be registered at most once with any particular selector.

Newly-created selectable channels are always in blocking mode.
Non-blocking mode is most useful in conjunction with selector-based
multiplexing. A channel must be placed into non-blocking mode before
being registered with a selector, and may not be returned to blocking
mode until it has been deregistered. (It seems that you can place it in
blocking before performing the io operation, and again to non blocking
after it)



SelectionKey register(Selector sel, int ops, Object att)

The key's interest set will have been changed to ops
If this channel is currently registered with the given selector then
the selection key representing that registration is returned.
If this method is invoked while another invocation of this method or of
the configureBlocking method is in progress then it will first block
until the other operation is complete. This method will then synchronize
on the selector's key set and therefore may block if invoked
concurrently with another registration or selection operation involving
the same selector.


**********************************************************************

Selector

A selector maintains three sets of selection keys:

* The key set contains the SelectionKeys representing the current
channel registrations of this selector. This set is returned by the
keys method.

* The selected-key set is the set of keys such that each key's channel
was detected to be ready for at least one of the operations identified
in the key's interest set during a prior selection operation. This set
is returned by the selectedKeys method.

* The cancelled-key set is the set of keys that have been cancelled
but whose channels have not yet been deregistered. This set is not
directly accessible.

A key is added to a selector's key set as a side effect of registering
a channel. The key set itself is not directly modifiable.

Keys are added to the selected-key set by selection operations. A key
may be removed directly from the selected-key set by invoking the set's
remove method or by invoking the remove method of an iterator obtained
from the set. Keys are never removed from the selected-key set in any
other way; they are not, in particular, removed as a side effect of
calls to select(..).

Selection is performed by the select(...), and involves three steps:

1. Each key in the cancelled-key set is removed from each key set of
which it is a member, and its channel is deregistered. This step leaves
the cancelled-key set empty.

2. The underlying operating system is queried for an update as to the
readiness of each remaining channel to perform any of the operations
identified by its key's interest set as of the moment that the
selection operation began. For a channel that is ready for at least
one such operation, one of the following two actions is performed:
1. if the channel's key is not already in the selected-key set then
it is added to that set and its ready-operation set is modified to
identify exactly those operations for which the channel is now
reported to be ready. Any readiness information previously recorded
in the ready set is discarded.
2. Otherwise the channel's key is already in the selected-key set,
so its ready-operation set is modified to identify any new operations
for which the channel is reported to be ready. Any readiness
information previously recorded in the ready set is preserved.

3. If any keys were added to the cancelled-key set while step (2) was
in progress then they are processed as in step (1).


Selectors are themselves safe for use by multiple concurrent threads;
their key sets, however, are not.

The selection operations synchronize on the selector itself, on the key
set, and on the selected-key set, in that order. They also synchronize
on the cancelled-key set during steps (1) and (3) above.

The close method synchronizes on the selector and all three key sets
in the same order as in a selection operation.

Changes made to the interest sets of a selector's keys while a selection
operation is in progress have no effect upon that operation; they will
be seen by the next selection operation.

Keys may be cancelled and channels may be closed at any time.
Hence the presence of a key in one or more of a selector's key sets does
not imply that the key is valid or that its channel is open.
Application code should be careful to synchronize and check these
conditions as necessary if there is any possiblity that another thread
will cancel a key or close a channel.
 
Jose Botella
Ranch Hand
Posts: 2120
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
And this is the testing code:
 
Jose Botella
Ranch Hand
Posts: 2120
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
And finally I wrote a testing code that sends and receives severals files, and simultaneously sends/receives user text messages. It uses a single thread and a selector.

Because the code is so long, I won't paste it here. It is on the 7th page of this thread enjoy yourself!
 
Jose Botella
Ranch Hand
Posts: 2120
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I hope you do not think I am a nagger but I have found the article Building Highly Scalable Java Servers with NIO ,by Nuno Santos, very interesting.
[ September 14, 2004: Message edited by: Jose Botella ]
 
Do the next thing next. That’s a pretty good rule. Read the tiny ad, that’s a pretty good rule, too.
a bit of art, as a gift, that will fit in a stocking
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic