Why are your pointers 64bit on a 32bit system?
What about pointer to pointer? I think you are missing a PLoad for that.
Missing function pointers (not castable to nativeint), ptrdiff and array access I think. Phantom types?
Have you thought about c++? Member pointers, method pointers?

I feel the type parameter 'a in 'a Ptr.t is deceiving as the API does not offer any type safety at all. I think type safety is better left to higher-level libraries like ocaml-ctypes.

@nojb The big advantage is that a higher-level library could do something like this:

type _ referent =
| Char : char referent
| Int32 : Int32.t referent
| Int64 : Int64.t referent
(* ... *)

let load (type 'a) (a : 'a referent) (ptr: 'a Ptr.t): 'a =
  match a with
  | Char -> Ptr.load8 ptr
  | Int32 -> Ptr.aligned_load32 ptr
  | Int64 -> Ptr.aligned_load64 ptr
  (* ... *)

@mrvn They are nativeints (so 32 bit on 32 bit system, 64 bit on a 64 bit system). Note that not all of C++'s pointer types make sense in OCaml, since its GC does not support interior pointers and this module has no notion of a struct.

On 64-bit systems I could potentially optimize on the basis that no existing CPU has a true 64-bit address space, thus I could store the pointer in a plain int and sign-extend the pointer before using it. That is an existing primop (%nativeint_of_int), so might be worthwhile – but it would make it much harder to pass pointers to foreign code without boxing or tagging, as the needed attributes would be different, and the 64-bit one not yet implemented in the compiler.

Function pointers are not supported because I don't know how to implement them efficiently in ocamlopt – this is my first time hacking on the compiler! Tips for how to do this (without calling out to a C function to actually invoke them) would be welcome.

Also, this is very incomplete – I have not implemented loads to pointers yet!

The Ptr.offset function allows for adding a pointer to a nativeint.

I deliberately did not implement pointer subtraction, because it is error-prone. You can implement it yourself if you wish:

let ptr_diff : 'a Ptr.t -> 'b Ptr.t -> nativeint = fun a b ->
  Nativeint.sub (Ptr.to_nativeint a) (Ptr.to_nativeint b)

and this is guaranteed to be correct.

Also, for FFI purposes I am wondering if a pointer and a C intptr_t are passed identically to C functions on all platforms ocamlopt targets. If so, then I can go ahead and add support for unboxing pointers when they are passed to FFI functions. I know that this is not true in general, and I believe this was the subject of a bug report to LLVM. I am also wondering if I can pass a C function an int32 when it expected a uint16_t or a uint8_t – if so, I can add support for untagging those types (after adding modules for them).

The end goal is to be able to bind C functions and data structures just like how Haskell's FFI does it: with the C data structures manipulated by OCaml code.

I still don't see why is it necessary/useful to have the phantom type. In your example you could instead use the signature

val load: 'a referent -> Ptr.t -> 'a

@nojb There the phantom type does help: it prevents someone from passing the wrong pointer type of pointer to the load operation. This proposal happens to ignore it – I plan on changing it to not ignore it.

I think for a first step you should keep things simple: remove the phantom type and move the functions to a submodule of Obj since the operations are unsafe.

• How do that compare to %caml_string_get*, %caml_string_get*u, %caml_string_set*, %caml_string_set*u.?
• Could one be implemented with the other?
• Do you plan to add functions for reading inside any Obj.t at some arbitrary offset: the goal is to read inside a custom block for example. You can't convert the Obj.t to a pointer Ptr.t since in that case it is not registered.
• How do that compare to the function for the bigarrays?

If thanks to this raw pointer view we can factorize some of these operations, I think it would be a great achievement in addition to support for C bindings in OCaml.

@bobot %caml_string_get*, %caml_string_get*u, %caml_string_set*, and %caml_string_set*u are exactly your third case: they read from any Obj.t (technically, any heap block) at any arbitrary offset (with bounds checking, for the versions that don't end in u).

Neither of these two can be implemented in terms of the other, at least not safely with respect to GC. The %caml_string* primitives assume that their argument is a well-formed OCaml value, which must be registered with the garbage collector. The %(aligned|unaligned)_load* primitives (and the upcoming %(aligned|unaligned)_store* primitives) assume that their argument is an OCaml nativeint that does not point into the garbage collected heap.

However, the bigarray and bigstring functions most definitely could be implemented in terms of the %caml_string_(set|get)* primitives and the new raw pointer support. They are implemented via an indirection through a Custom_tag block – which means that it is safe to access the fields of the block from OCaml via the %caml_string_(set|get)* primitives, since the garbage collector will not change them. Once the field that contains the actual heap primitive is obtained, the %(aligned|unaligned)_(load|store)* primitives can manipulate the actual array (allocated on the C heap) that stores the bigarray/bigstring. In fact, I suspect that this would be a big performance win: by splitting the functions into separate primitives, the body of the functions becomes open to inlining. This in turn should allow for CSE and lifting of one layer of indirection (the Custom block) out of inner loops.

As far as I remember, the unsafe versions of the string get/set functions do not make any assumptions about the structure of the value.

I would also suggest that if writing a raw pointer module, performance should be a prime concern, which means not representing pointers as boxed values. There are reasonable ways of representing them that do not require this, for example setting the bottom bit on a suitably-aligned pointer, and using %int_as_pointer to translate from that representation to the "real" pointer.

Please keep the phantom type. Otherwise any pointer type can be passed as any other pointer type to a function expecting one.

@DemiMarie The problem with function pointers is that they may (and are not) be the same size as data pointers. They do not fit into a value. A function pointer needs to be boxed.

As for misusing intptr_t to pass pointers I think that would fail on at least m68k (does ocaml support m68k?) since addresses are passed in A0/A1 while integers are passed in D0/D1 as far as I remember from AmigaOS. Why don't you want to cast pointers to (void*)? Have you looked at Ctypes and how it does it?

@mrvn The reason for wanting to misuse intptr_t to pass pointers (if it works!) is because OCaml can already pass intptr_t values to C functions directly, while passing void* pointers would require additional modifications to the compiler.

Also, on what architectures do function pointers not fit in a nativeint, that are 32 or 64 bit?

@mshinwell The problem is that C pointers, unlike OCaml pointers, cannot be assumed to be aligned. On some 32-bit systems, user code can use 3GB of address space, so a pointer can use all 32 bits. On 64 bits, sign extending a 63-bit value can point to the entire address space.

@DemiMarie I thought function pointers where always larger than void* but I was confusing them with c++ method pointers (which need extra space to handle virtual table lookup). Cases where function pointers differ from data pointers are when you have segmented memory like on MSDos. I don't know what other architectures ocaml supports but google shows there is an ocaml for MSDos.

As for boxing pointers: Could we have a [@@Aligned] attribute that would keep them unboxed?

OCaml does not support 16 bit machines.

The ideal solution to boxing of pointers (and of other types) is
whole-program monomorphization (as used by MLton and Rust), reified
generics (.NET), or at least explicit unboxing (GHC Haskell) but since
OCaml won't be getting any of these anytime soon I am not sure what to do.
OCaml's use of tagged integers representation is the real problem here.

As far as workarounds, I am not sure how to make the representation of a
type depend on an attribute (this is my first time trying to contribute to
the OCaml compiler!!!). Another approach is to always unbox pointers on
64-bit and when not targeting x86 (under the assumption that performance on
legacy x86-32 systems doesn't matter, and that using more than 2GiB on
32-bit ARM/MIPS/POWER/SPARC is forbidden by the OS and/or hardware).

I could have a separate type for pointers that are guaranteed to be 16-bit
aligned. But I don't want to just blindly lop of the lowest bit, because
this will likely cause random data corruption (not even, necessarily, a
crash) that will be extremely difficult to debug, and the check is
extremely cheap (an AND and a branch that should always be successfully
predicted by the CPU). Instead, I would write conversion functions that
throw Invalid_argument on failure (to indicate that the caller has a bug).
On Jul 7, 2016 6:07 AM, "Goswin von Brederlow" notifications@github.com
wrote:

@DemiMarie https://github.com/DemiMarie I thought function pointers
where always larger than void* but I was confusing them with c++ method
pointers (which need extra space to handle virtual table lookup). Cases
where function pointers differ from data pointers are when you have
segmented memory like on MSDos. I don't know what other architectures ocaml
supports but google shows there is an ocaml for MSDos.

As for boxing pointers: Could we have a [@@Aligned
https://github.com/aligned] attribute that would keep them unboxed?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#652 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AGGWBzJdsAEdVslKvSDOZMt5IvmwJKSPks5qTM_6gaJpZM4JDx25
.

@DemiMarie I think you should have one type for 16-bit aligned pointers (int) and one for misaligned pointers (nativeint). Almost all pointers will be aligned to 16-bits (actually 32 bits) on most platforms AFAIK, with the exception of char*. Even in structs, you'll get padding unless you specifically disable it (with attribute((packed))).

Also, why do you have options for loading Pint32/64/nativeint? AFAIK all platforms nowadays match their nativeint size to the pointer size.

Will implement (along with writes to pointers and possibly other
operations, such as an OCaml interface to Custom blocks).
On Jul 7, 2016 10:47 PM, "Yotam Barnoy" notifications@github.com wrote:

@DemiMarie https://github.com/DemiMarie I think you should have one
type for 16-bit aligned pointers (int) and one for misaligned pointers
(nativeint). Almost all pointers will be aligned to 16-bits (actually 32
bits) on most platforms AFAIK, with the exception of char_. Even in
structs, you'll get padding unless you specifically disable it (with
*attribute_((packed))).

Also, why do you have options for loading Pint32/64/nativeint? AFAIK all
platforms nowadays match their nativeint size to the pointer size.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#652 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AGGWB67Y4eke8NvakC0T4OcYAgHn7zkSks5qTbo0gaJpZM4JDx25
.

For fun I played to access bigarray using Obj.Pointer

module A1 = Bigarray.Array1

external get_custom_64: Obj.t -> int -> Int64.t =  "%caml_string_get64u"

let () =
  let a = A1.create Bigarray.int64 Bigarray.c_layout 10 in
  let get_c_field x n = get_custom_64 (Obj.repr x) ((n+1)*8) in
  let get_c_array x = Int64.to_nativeint (get_c_field x 0) in
  let get_c_int64 x n =
    Obj.Pointer.unaligned_load64
      Nativeint.(add (get_c_array x) (of_int (n * 8)))
  in
  A1.set a 2 6L;
  Printf.printf "%Lu\n%!" (get_c_int64 a 2)

Remaks:

• a make depend is missing, in stdlib/.depend obj.cm* doesn't depend on nativeint.cm*
• I think it is easier for such low level feature if Obj.Pointer.t is externally an alias to nativeint, or that the function directly use nativeint.
• Somewhere it must be specified that the memory is byte-addressable. In C it is transparent since the addition of an offset use the size of the pointed elements, here it is not.

A C pointer has 2 hidden attributes:

1. alignment
2. target size
   Maybe this could be added as phantom types:

type align = Align: int -> align
type size = Size: int -> size 
type ('a, align, size) t

I don't think we should bother to unbox pointers. When one knows that a pointer is 2-aligned, they can always convert it to an int to pass to it around, then convert it back to a nativeint before using the primitives. The compiler will be able to get rid of the temporary boxing

I thought a bit more about this, I think we should do the same as for the string/bigstring primitives: just add compiler support - i.e. the primitives - and let users write their own API with phantom types if they want.

@DemiMarie do you think you can finish the primitives, and remove the stdlib additions? I'll then review the patch and merge it. We've have a use case at Jane Street where using the string primitives wouldn't be safe so I'd like to have this feature in.

Can you also add some tests for these primitives? You can copy the tests for the string primitives in testsuite/tests/prim-bigstring/string_access.ml

@bobot, I'm not sure at all about the Obj.Repr module. In any case it should be a separate PR as it's not related to raw foreign pointers. Factorization seems good, but I tend to think we should do it in a separate PR to minimize the diff to review to get this in.

What I'm going to do is start from your branch, cleanup a bit the history, cleanup the bytecode stubs (they should be shared with the string stubs) and add some tests, then submit a new PR

What I'm going to do is start from your branch, cleanup a bit the history, cleanup the bytecode stubs (they should be shared with the string stubs) and add some tests, then submit a new PR

@diml seems good, and a worthwhile cleanup with @DemiMarie technique. The last cleanup would be, once all these PR are accepted, to remove the bigstring primitive and code all directly in OCaml.

Also @diml you can change a bit the primitives to assume a nativeint as index argument. This could allow to remove some untagging in some cases. The usual way to use them would probably still be Ptr.load (Nativeint.of_int n), but some specific cases might benefit from that.

By the way I now consider it a mistake to have used int as argument of string_*_set/ref, if you want to change that, you have my blessing. (but notify me so that I can update ocplib-endian)

@bobot, If we implement some bigarray primitives in OCaml, we need to be quite careful to have some ways to keep references to the bigarray value alive while the bigarray memory is used:

let ba_get ba i =
  let data = Ptr.loadnative ba 0 in
  Ptr.loadnative data i

let iter ba f =
  for i = 0 to Array1.dim a - 1 do
    f (ba_get ba i)
  done

let stuff ba =
  let f x = Sys.opaque_identity (x, x) in
  f ba_get ba 0;
  iter ba f

Supposing that everything is inlined, f will allocate, but won't invalidate pointer loads, hence all the let data = Ptr.loadnative ba 0 in could be shared (it won't happen with the current OCaml, but this is not a big supposition) and if no other reference to ba exist, it could be dead when entering the loop, hence might be collected.

This means that we need some way to tell that data maintains ba alive. This could be achieved by adding some additional argument to the load primitive to tell that some other value must be alive for this primitive to be valid.

This means that we need some way to tell that data maintains ba alive. This could be achieved by adding some additional argument to the load primitive to tell that some other value must be alive for this primitive to be valid.

Could we have another more general way? In tests for the GC, it often happen that we also want to keep a value alive for some time and to let it go later. We could have something like:

external keep_alive: 'a -> (unit -> 'b) -> 'b = "%keep_alive"
(**  [keep_alive handled f] ensure that [handled] is alive until the
      end of the computation of [f]. It returns the result of [f]. *)

@bobot for the user facing part yes, but we would still need something else to represent that in Mach.

Should I change the documentation of Obj? The current documentation says "Operations on the internal representation of values" which this certainly isn't – it is about operations on C data structures, and could just as easily be implemented by a different OCaml backend that used a totally different representation of heap objects. Code using it would still be portable to such a backend (that implemented the primitives). For instance, OCamlJava could use sun.misc.Unsafe/jdk.internal.misc.Unsafe, and a hypothetical .NET backend could use .NET's raw pointer support.

@bobot I don't like the keep_alive function since nothing ensures it actually gets used.

I think at least for bigarray (and similar structure access) it should be impossible to screw this up by accident. For that I think we need a temporary object that contains a) the original pointer to the bigarray block, b) the offset into the bloc (0 here). And then a function to load from an indirect pointer, e.g. Ptr.loadnativeref, that takes the temporary object and returns the data.

A more general form could be to store the pointer to the ocaml value keeping the data alive + C pointer to the data.

@mrvn said:

@bobot I don't like the keep_alive function since nothing ensures it actually gets used.

That's true but only the developers of the bigarray library must not forget to use it. Raw pointers are already not for the casual users and very unsafe, it should be used indirectly only behind nice API (ex:bigarray), or with the help of library similar to ctypes. So I don't see the need to over-engineer the API provided by the compiler.

PS: just for simplifying keep_alive and giving it a semantic., it can of course already be defined (in an inefficient way). It is similar to ignore, but can't be removed:

external keep_alive: 'a -> unit = "keep_alive" [@@noalloc]

value keep_alive(value x){
    return Val_unit;
}

value keep_alive(value x){
    return Val_unit;
}

This is what ctypes currently does. It'd be handy to have support in the standard library.

How does external keep_alive: 'a -> unit = "keep_alive" [@@noalloc] keep the value alive at all? I don't think you can simplify it to that extend. You can define part of it as keep_alive_helper like that and then use:

let keep_alive x fn =
  match fn () with
  | res -> keep_alive_helper x; res
  | exception exn -> keep_alive_helper x; raise exn

Have you considered passing x to fn? If x needs to remain alive while fn runs then it seems verry likely it will have to access it as a starting point. I imagine a lof of code would look like this:

keep_alive x (fun () -> fn x)

How does external keep_alive: 'a -> unit = "keep_alive" [@@noalloc] keep the value alive at all?

It's a use of the value that can't be optimized away.

Whenever keep_alive v appears in a program you can be confident that v is still live at that point.

The 'efficient' version of keep_alive is quite difficult to do: we might want to be able to eliminate it if the result of the access is not used, hence the proposition of merging it with the read and write primitives.

Is there a reason to keep this PR open, considering #724?

I suppose there is no need to have both PRs