Internal`CopyListStructure does not produce packed array output from packed array input

Question

Consider this example:

array = {{1, 2, 3}, {4, 5, 6}, {7, 8, 9}};
array // Developer`PackedArrayQ
(* False *)

packedarray = Developer`ToPackedArray[array];
packedarray // Developer`PackedArrayQ
(* True *)

arrayCS = Internal`CopyListStructure[array, Flatten@array];
arrayCS // Developer`PackedArrayQ
(* False *)

packedarrayCS = 
  Internal`CopyListStructure[packedarray, Flatten@packedarray];
packedarrayCS // Developer`PackedArrayQ
(* False *)

So, my question is: Is there an option to make Internal`CopyListStructure to output packed array given packed array data as input, so that one doesn't have to do one more Developer`ToPackedArray over the output?

Update

Although my original question stands up there as it is, here'st clarification per comments why I Map a RuntimeAttributes -> {Listable} function over the dataset.

My actual input is not a rectangular array, so I don't use ArrayReshape for the same reason.

Building on example by @b3m2a1 - here's the way it works in case the input data is of rectangular shape:

makeGrid[t1_, t2_, t3_, t4_, t5_, cf_, cfMap_, ar_, unfl_, ve_] := 
  Module[{data, outputOK, packedArrayQ},
   outputOK = SameQ[cfMap, #] & /@ {cf, cfMap, ar, unfl, ve};
   packedArrayQ = Developer`PackedArrayQ /@ {cf, cfMap, ar, unfl, ve};
   data = 
    SortBy[Join[{{"Listable Function @", t1}, {"Listable Function /@",
         t2}, {"Flatten>>ArrayReshape", 
        t3}, {"Flatten>>Internal`CopyListStructure", 
        t4}, {"Vectorized", t5}}, Partition[outputOK, 1], 
      Partition[packedArrayQ, 1], 2], #[[2]] &];
   Text@Grid[
     Prepend[data, {"Approach", "RepeatedTiming", "SameQ", 
       "PackedArrayQ"}], 
     Background -> {None, {Lighter[Yellow, .9], {White, 
         Lighter[Blend[{Blue, Green}], .8]}}}, 
     Dividers -> {{Darker[Gray, .6], {Lighter[Gray, .5]}, 
        Darker[Gray, .6]}, {Darker[Gray, .6], 
        Darker[Gray, .6], {False}, Darker[Gray, .6]}}, 
     Alignment -> {{Left, Right, {Left}}}, ItemSize -> {{22, 10}}, 
     Frame -> Darker[Gray, .6], ItemStyle -> 14, 
     Spacings -> {Automatic, .8}]
   ];

cf1 = Compile[{{x, _Real}}, x^2, RuntimeAttributes -> {Listable}];

rr3 = RandomReal[{}, {115, 222, 30}];

{t1, cf} = cf1[rr3] // RepeatedTiming;

{t2, cfMap} = cf1 /@ rr3 // RepeatedTiming;

{t3, blank} = RepeatedTiming[
   flrr3 = Flatten@rr3;
   drr3 = Dimensions@rr3;
   ar = ArrayReshape[cf1[flrr3], drr3];
   ];

{t4, blank} = RepeatedTiming[
   flrr3 = Flatten@rr3;
   unfl = Internal`CopyListStructure[rr3, cf1[flrr3]];
   ];

{t5, ve} = rr3^2 // RepeatedTiming;

makeGrid[t1, t2, t3, t4, t5, cf, cfMap, ar, unfl, ve]

So, it holds true that vectorized approach is the fastest and so forth.

However, this is what it looks like in case the data is not rectangular-shaped:

rr3[[1, 1, 1]] = Nothing;

{t1, cf} = cf1[rr3] // RepeatedTiming;
{t2, cfMap} = cf1 /@ rr3 // RepeatedTiming;
{t3, blank} = RepeatedTiming[
   flrr3 = Flatten@rr3;
   drr3 = Dimensions@rr3;
   ar = ArrayReshape[cf1[flrr3], drr3];
   ];
{t4, blank} = RepeatedTiming[
   flrr3 = Flatten@rr3;
   unfl = Internal`CopyListStructure[rr3, cf1[flrr3]];
   ];
{t5, ve} = rr3^2 // RepeatedTiming;
makeGrid[t1, t2, t3, t4, t5, cf, cfMap, ar, unfl, ve]

What are you using this for? There may be a more idiomatic workaround that preserves the packed array — b3m2a1, Commented Aug 21 at 16:52
@b3m2a1 For large nested lists, instead of mapping a listable function, flatten the list > feed it to the listable function > unflatten the result; also, to delete multiple-level positions in lists with depth>2: flatten the list > use Part to assign Nothing > unflatten back. — Anton, Commented Aug 21 at 17:36
Why not f[nestedList], if f is Listable? If you're going to be applying a listable function to the bottom level, it might be good to leave things unpacked (f[packed] yields unpacked, if Listable). [Your comment makes an example use-case seem important.] — Michael E2, Commented Aug 21 at 17:48
What do you mean by "the listable function is applied to every sub-list of the list"? A Listable function is ultimately applied only to non-List elements at the bottom levels of a nested list and not to the sublists. By "ultimately applied," I mean the function is evaluated. F[{a, {b}}] becomes {F[a], {F[b]}} before any definition-rules for F[] are used. Can you give a similar concrete example where my claims seem to be wrong? — Michael E2, Commented Aug 21 at 18:29
(1) Is your actual function vectorized? That would be a significant fact that wasn't in the OP. #^2 & is rather a special choice for a test function. (2) In the OP, the original array was not packed; does the data start in packed or unpacked form? You should including the time to pack it, if not. Vectorized loses some of its advantage then. (3) You should use (or try) the option Parallelization -> True in cf1. — Michael E2, Commented Aug 22 at 13:30

b3m2a1 · Accepted Answer · 2025-08-21 18:36:04Z

Per the comment, here's another way to do the reshapes in packed fashion which I think achieves what you want using the builtins

pa = RandomReal[{}, {5 , 2, 3}];

res = Flatten@RandomReal[{}, Dimensions@pa]; (* pretend this came out of a flattened listable compiled function *)
ArrayReshape[res, Dimensions@pa] // Developer`PackedArrayQ

True

packedarrayCS = Internal`CopyListStructure[pa, res];
packedarrayCS // Developer`PackedArrayQ

False

packedarrayCS == ArrayReshape[res, Dimensions@pa]

True

Here's a more explicit use case to show that there might be a small benefit to this flattening

woof = Compile[{{x, _Real}}, x^2, RuntimeAttributes -> {Listable}];

pa = RandomReal[{}, {115, 222, 30}];

woof[pa] // RepeatedTiming // {#[[1]], Developer`PackedArrayQ[#[[2]]]} &

{0.030855, True}

fpa = Flatten@pa;
dpa = Dimensions@pa;

ArrayReshape[woof[fpa], dpa] // 
  RepeatedTiming // {#[[1]], Developer`PackedArrayQ[#[[2]]]} &

{0.0239654, True}

Obviously this has nothing on vectorization when applicable

pa^2 // RepeatedTiming // {#[[1]], Developer`PackedArrayQ[#[[2]]]} &

{0.000563136, True}

If I include flattening in the timing of the second method, the advantage disappears. — Michael E2, Commented Aug 21 at 18:40
@MichaelE2 makes sense, I don't really see why you'd want to do the flattening in general, but the OP seems to have a reason for it (maybe they have a different function that returns things flattened) — b3m2a1, Commented Aug 21 at 19:06

Jason B. · Accepted Answer · 2025-08-22 12:23:02Z

4

Is there an option to make Internal`CopyListStructure to output packed array given packed array data as input

No:

In[6]:= Options[Internal`CopyListStructure]

Out[6]= {}

Which makes a lot of sense. If your original list is ArrayQ then CopyListStructure is the wrong method to use. You would use CopyListStructure when copying data into a ragged list of lists.

If you want CopyListStructure to return a packed array for some reason, just use something like this:

myCopyListStructure = Developer`ToPackedArray @* Internal`CopyListStructure

edited Aug 22 at 12:23

answered Aug 21 at 14:04

Jason B.

71.9k3 gold badges151 silver badges313 bronze badges

1

$\begingroup$ I can't actually tell what point you are making with your post, it seems off topic with respect to the question at hand. Also, if your list isn't ArrayQ then why are you concerned about whether it is packed? $\endgroup$
– Jason B.
Commented Aug 22 at 12:52
$\begingroup$ I'm not making any point, the question is in the headline. If you feel that the question is off topic, I don't see why you don't flag it as such. $\endgroup$
– Anton
Commented Aug 22 at 13:03

Add a comment |

Stack Exchange Network

Internal`CopyListStructure does not produce packed array output from packed array input

Update

2 Answers 2

Hot Network Questions

Internal`CopyListStructure does not produce packed array output from packed array input

Update

2 Answers 2

Related

Hot Network Questions