Current implementation of EdgeView on EVE devices has build in limitations in operation:

  • In the EVE log search, the maximum time range is 5 hours
  • When use log/copy-logfiles, which copies all the log files in a time range to the local computer, has a maximum range of 30 minutes
  • When use tar command, the maximum file size in the directory is 500 MBytes

Those were intended to limit the processing and network activities on the EVE devices. Those limitations can cause some inconvenience for the EdgeView usage and user experience issues. Some users asked to remove those EdgeView limitations.

A related issue to this file size limitation of transferring through the network is the bandwidth usage of the EdgeView Dispatcher server. Heavy traffic from multiple devices can have negative impact on the Dispatcher's service on the cloud side, unless the dispatcher server is customer owned or a private one.


Proposed Solution on EdgeView

  • Remove all the above limitations
  • Impose a rate-limit, 5 Mbps per EdgeView session, on the EVE device side
  • Dispatcher can be configured to send a message to device to disable the rate-limit implementation on EdgeView, in the case of a private dispatcher


Implementation

The removal of the size and range limitations goes together with the rate-limit implementation on the same EVE release.

Rate-Limit on EdgeView Traffic

The implementation needs to handle two separate tasks in the rate-limit of EdgeView traffic:

  1. The file copy and transfer, which includes copy files, tar a directory and transfer the tar-file
  2. TCP sessions, EdgeView user can use 'scp' to do large file transfer from Applications or host to local laptop


For the first one, this is to do with the operation in EdgeView coping a file from the device to the local laptop. Before the copy operation, it knows the total size, and it knows the rate needs to be applied. Each chunk of the data transfer, a fixed time delay can be inserted to achieve the whole file transfer rate.

For the second one, the 'golang.org/x/time/rate' is used to apply to the TCP session data transfer. The TCP inbound on EdgeView session data is read one chunk at a time, and it is relayed to the websocket session towards the dispatcher, the rate can be calculated and time delay can be obtained through each TCP data transfer from the device.

The EdgeView client software on the laptop can print out the transfer rate after it receives the completion of EdgeView file transfer. In the case of the user launches two sessions to the same device, one to copy a large size file, the other is using a TCP session to transfer data, the rate of the EdgeView from the device can be twice of the limit intended, but this should be a rare case, and the EdgeView rate-limiting is not meant to be absolute accuracy.

Disable Rate-Limit Mechanism

The implementation can also listen to the Dispatcher message on websocket connection for disabling the rate-limit. This can be used in the future case where the dispatcher is a private one and it does not care about the amount of data being used by EdgeView. The configuration of this mechanism on dispatcher is out side the scope of this proposal.


  • No labels

2 Comments

  1. Erik Nordmark

    I thought the current limits was also there to limit the amount of temporary space needed for e.g., tar files.

    Will the new implementation use streaming APIs to avoid creating any temporary tar (or other) files?

    1. Naiming Shen

      Good point. No, the new implementation still needs to save into a tar-file. But we can check if there is enough free space on the disk before allowing the tar operation on a directory, since we do check the size first.