![]() ![]() The basic logic is to compute the parameters for splitting the array and then use array_split to split the array along each axis (or dimension) of the array. This can be done so that the resulting arrays have shapes slightly less than the desired maximum or so that they have exactly the desired maximum except for some remainder at the end. The answer and comments suggests switching to 64 bit (which I already have) and using mmap. ![]() I also saw this question, since it's actually my exact problem. This question (among other similar ones) has answers explaining how to split into a certain number of chunks, but this does not explain how to split into a certain size. I have seen this function that uses reshape, but it doesn't work if the number of rows and columns doesn't divide the size exactly. ![]() I'm using Python 3.6 64-bit with numpy 1.14.3 (latest). How can I split this into the desired chunks? Since this is just to avoid errors with extremely large arrays, it's okay if the result isn't exact. Let's also say I want them in chunks of no larger than 50 by 50. My data isn't necessary possible to split evenly, and it's definitely not guaranteed to be splitable by the size I want. (I can aggregate the results after the operation runs on each piece.) The fact that my problem is MemoryErrors means that it's important that I can cap the size of the arrays somehow, rather than split them into a constant number of pieces.įor an example, let's generate a 1009 by 1009 random array: a = (, (1009,1009)) Because some operations I need to perform throw MemoryErrors, I would like to try splitting the array up into chunks of a certain size and running them against the chunks. One data set is 55732 by 257659, which is over 14 billion elements. I have some very large two-dimensional numpy arrays. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |