Alpakka and S3 truncating downloaded files

@fsoler wrote:

Hello,

I have a simple piece of code, based on alpakka examples, which should download some file from S3 for further processing:

S3.download(bucket, file)
  .runWith(Sink.head)
  .flatMap {
    case Some((data, _)) =>
      data.map(_.utf8String).runWith(Sink.head).map(Some.apply)
    case None =>
      Future.successful(None)
    }

The problem is that the file content is getting truncated, the file size from the ObjectMetadata is correct it has ~2M it isn’t a huge file.

What I noticed is, when I use Sink.head the file content is from the beginning to middle if I change it to Sink.last it is from the middle to the end. Am I getting chunks from the file but why are they not seem to be streamed?

Can’t figure out what’s happening and how to fix this.

This is the log after the download:

2020-03-18 17:10:09,390 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (WaitingForResponse)] Received response
2020-03-18 17:10:09,392 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (WaitingForResponse)] Before event [onResponseReceived] In state [WaitingForResponse] for [19 ms]
2020-03-18 17:10:09,393 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (WaitingForResponse)] onResponseReceived in WaitingForResponse with false
2020-03-18 17:10:09,395 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (WaitingForResponseDispatch)] After event [onResponseReceived] State change [WaitingForResponse] -> [WaitingForResponseDispatch]
2020-03-18 17:10:09,395 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (WaitingForResponseDispatch)] Before event [onResponseDispatchable] In state [WaitingForResponseDispatch] for [0 ms]
2020-03-18 17:10:09,396 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (WaitingForResponseEntitySubscription)] After event [onResponseDispatchable] State change [WaitingForResponseDispatch] -> [WaitingForResponseEntitySubscription]
2020-03-18 17:10:09,396 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (WaitingForResponseEntitySubscription)] Before event [onResponseEntitySubscribed] In state [WaitingForResponseEntitySubscription] for [10 ms]
2020-03-18 17:10:09,396 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (WaitingForEndOfResponseEntity)] After event [onResponseEntitySubscribed] State change [WaitingForResponseEntitySubscription] -> [WaitingForEndOfResponseEntity]
2020-03-18 17:10:09,396 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (WaitingForEndOfResponseEntity)] Before event [onResponseEntityCompleted] In state [WaitingForEndOfResponseEntity] for [1 ms]
2020-03-18 17:10:09,397 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (Idle)] After event [onResponseEntityCompleted] State change [WaitingForEndOfResponseEntity] -> [Idle]
2020-03-18 17:10:09,397 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (Idle)] Slot became idle... Trying to pull
2020-03-18 17:10:09,398 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (Idle)] Connection cancelled
2020-03-18 17:10:09,398 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (Idle)] Connection completed
2020-03-18 17:10:09,398 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (Idle)] Before event [onConnectionCompleted] In state [Idle] for [3 ms]
2020-03-18 17:10:09,398 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (ToBeClosed)] After event [onConnectionCompleted] State change [Idle] -> [ToBeClosed]
2020-03-18 17:10:09,399 DEBUG[S3Downloader/Pool(shared->http://localhost:5678)] a.h.i.e.client.PoolGateway:97 - [0 (ToBeClosed)] State change from [Idle] to [ToBeClosed]. Closing the existing connection.

Thanks

Posts: 1

Participants: 1

Read full topic