Technology

59374 readers

3767 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

[email protected]

Efficiently Manage Memory Usage in Pandas with Large Datasets (geekpython.in)

submitted 3 months ago by [email protected] to c/[email protected]

7 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 3 points 3 months ago (5 children)

Is there a benefit to doing CoW with Pandas vs. offloading it to the storage? Practically all modern storage systems support CoW snaps. The pattern I'm used to (Infra, not big data) is to leverage storage APIs to offload storage operations from client systems.

[–] [email protected] 2 points 3 months ago (2 children)

If you are doing data processing in pandas CoW allows to avoid of a lot of redundant computations on intermediate steps. Before CoW any data processing in Pandas required manual and careful working with code to avoid the case described in the blog post. To be honest I cannot imagine the case of offloading each result of each operation in the pipeline to the storage…

[–] [email protected] 2 points 3 months ago (1 children)

So you would be using CoW in-memory in this case?

[–] [email protected] 1 points 3 months ago

If I already use Pandas for processing my data in-memory, CoW can significantly improve the performance. That was my point.

load more comments (2 replies)