I think clojure's array-maps do it but with some limitations:

https://clojure.org/reference/data_structures#ArrayMaps

I realized belatedly that ordering matters for my application. . .

Even looked at Assocative Containers[1] before I thought, "alist!"

In the last ten years I've only needed ordered maps once or twice. In one case it was for a custom query language I wrote that generated queries from data alone.

eg. (query db :users (array-map :gender "female" :name "xena"))

so in this example adding the :gender clause first would restrict the query and improve performance.

I also remember, in my early clojure days, I made the mistake of relying on standard hash-maps:

eg. (query db :users {:gender "female" :name "xena"})

Until I discovered a query where the performance died and found out hash-maps are actually array-maps (under the hood) until 9 entries. It then auto converts to real hash-maps and loses its order (Clojure does this because maintaining order on bigger data sets is costly from an efficiency/memory perspective).

  (= tpipe {todo 
              '({id 1 cont "get eggs" done '(nil)}
                {id 23 cont "fix toilet" done '(nil)})
            week '(nil)
            today '({id 23 cont "Build something that works in arc" done '(nil)})
            done '({id 23 cont "Research Ordered Associative Arrays" done '(2019 1 21)})})

  (= apipe '((todo ({id 1 cont "get eggs" done '(nil)} 
                    {id 23 cont "fix toilet" done '(nil)})) 
             (week (nil)) 
             (today ({id 23 cont "Build something that works in arc" done '(nil)}))
             (done ({id 23 cont "Research Ordered Associative Arrays" done (2019 1 21)}))))

  arc> ((alref apipe 'todo) 0)
  '(%braces id 1 cont "get eggs" done '(nil))

  arc> tpipe!todo.1
  '(%braces id 23 cont "fix toilet" done '(nil))

  arc> tpipe!todo.1!id

  arc> ((alref apipe 'todo) 0) 'id)

    (= npipe {todo 
                (list {id 1 cont "get eggs" done '(nil)}
                      {id 23 cont "fix toilet" done '(nil)})
              week '(nil)
              today (list {id 23 cont "Build something that works in arc" done '(nil)})
              done (list {id 23 cont "Research Ordered Associative Arrays" done '(2019 1 21)})})

    (= unqpipe {todo 
                `(,{id 1 cont "get eggs" done '(nil)}
                  ,{id 23 cont "fix toilet" done '(nil)})
                week '(nil)
                today `(,{id 23 cont "Build something that works in arc" done '(nil)})
                done `(,{id 23 cont "Research Ordered Associative Arrays" done '(2019 1 21)})})

Ideal world with key and index access:

  arc> pipe!todo.1!id
  23

  arc> pipe.0.1!id
  23

Quotation is helpful for metaprogramming in a tangible way, because we can easily refactor code in ways that move parts of it from being quoted to being unquoted or vice versa.

And quotation is limited to metaprogramming in a tangible way, because we can only quote data that's reasonably maintainable in the same format we're maintaining our other code in. For instance, an Arc `quote` or `quasiquote` operation is itself written inside an Arc source code file, which is plain text, so it isn't very useful for quoting graphics or audio data.

We can of course use other functions or macros to construct those kinds of data. That's essentially Arc's relationship with tables. When we've constructed tables, we've just used (obj ...) and (listtab ...) and such.

Adding tables to the language syntax is doable, but it could have some quirks.

  ; Should this cause an error, or should it result in the same thing as
  ; '(let i 0 `{,++.i "foo"}) or '(let i 0 `{,++.i "foo"})? Each option
  ; is a little surprising, since any slight edit to the code, like
  ; replacing one ++.i with (++ i 1), would give us valid code to
  ; construct a two-entry table, and this code very well might have
  ; arisen from a slight edit in the opposite direction.
  '(let i 0
    `{,++.i "foo" ,++.i "bar"})
  
  ; Should this always result in 2, or should it result in 1 if "foo"
  ; comes last in the table's iteration order?
  (let x 0
    `{"foo" ,(= x 1) "bar" ,(= x 2)}
    x)

A macroexpander needs to extract two things from the code at any given time: The name of the next macro to expand, and the region of syntax the macro should operate on. Symbols help encode macro names. Lists and symbols together help encode regions of plain text. I think it's for these reasons that symbols and lists are so essential to Arc's syntax.

Arc has other kinds of data that commonly occur in its quotable syntax, like strings and numbers, but it doesn't need them nearly as much as symbols and lists. Arc could expand symbols like |"Hello, world!"| and |123| as ssyntax, or it could simply let everyone put up with writing things like (string '|Hello, world!|) and (int:string '|123|) each time.

> I recommend not expecting `quote` or `quasiquote` to be very useful outside the context of metaprogramming.

My immediate reaction is to disagree. A lot of the reason Lisp is so great is that quasiquotation is orthogonal to macros/metaprogramming.

    > ; Should this cause an error, or should it result in the same thing as
    > ; '(let i 0 `{,++.i "foo"}) or '(let i 0 `{,++.i "foo"})?

In general it feels unnecessarily confusing to include long doc comments in code fragments here. We're already using prose to describe the code before and after.

Code comments make sense when sharing a utility that you expect readers to copy/paste directly into a file to keep around on their disks. But I don't think that's what you intend here?

I think they're mostly unrelated topics, but I was using the quirks of tables in `quasiquote` to motivate keeping the number of quasiquotable syntaxes small and focused. Since I believe quotation is essentially only good for code generation (as I explain in more detail in the other reply), my preference is generally to focus the quasiquotable syntaxes on that purpose alone.

---

"In general it feels unnecessarily confusing to include long doc comments in code fragments here. We're already using prose to describe the code before and after."

Sorry, and thanks for the feedback on this.

There's a deeper problem here where my posts can get a bit long, with a lot of asides. :) I thought of those code examples as an aside or a subsection. If you were going to skim over the code, I wanted it to be syntactically easy to skim over the related prose at the same time.

This was something I felt was particularly worth skipping over. Ultimately, the quirks of using tables as syntax are mostly just as easy to put up with as the quirks of using tables for anything else. (I've gone to the trouble to make what I think of as non-quirky tables for Cene, but it's a very elaborate design, and I wouldn't actually expect to see non-quirky tables in Arc.)

Since I was only using these quirks to motivate why `quasiquote` would tend to be focused on code generation, I probably didn't invest enough space to fully explain what the quirks were. I'll try to explain them now....

---

"Those two fragments are the same?"

Whoops, those two fragments were supposed to be '(let i 0 `{,++.i "foo"}) and '(let i 0 `{,++.i "bar"}).

---

"Finally, both your examples seem to be more about side effects in literals? That is a bad idea whether it's a table literal or not, and whether it uses quasiquoting or not. Do you have a different example to show the issue without relying on side-effects?"

I don't know if I'd say the unquoted-key example depends on side effects, but the unquoted-value example very much does. Here it is again:

  (let x 0
    `{"foo" ,(= x 1) "bar" ,(= x 2)}
    x)

Removing side effects from the language is a great remedy for this, but typically that kind of effort can only go so far. In an untyped language, we usually have to deal with the side effects of run time type errors and nontermination, even if we eliminate everything else:

  `{key1 ,(accidentally-cause-a-run-time-error) key2 ,(loop-forever)}

  `{key1 ,(compute-with-64TB-of-space) key2 ,(compute-for-800-years)}

If the programmer actively intends for this program to fail fast, you and I will probably agree they would be better off sequencing the operations a little more explicitly, maybe like this:

  (let val1 (compute-with-64TB-of-space)
    `{key1 ,val1 key2 ,(compute-for-800-years)})

That's when they have to realize that the tables in their syntax have gotten in the way of this guarantee.

Simple solution: We clearly document this so people don't expect left-to-right evaluation order in this situation.

Alternative simple solution: We make tables order-preserving so they can be evaluated as expected.

That covers the unquoted-value example.

Now let's consider the unquoted-key example:

  '(let i 0
     `{,++.i "foo" ,++.i "bar"})

While it looks like this example depends on side effects (in this case mutation), I'm not so sure it does. Here's an alternative example which shows the same issue without necessarily using side effects:

  '`{,(current-location) "foo" ,(current-location) "bar"}

Is it a side effect? Maybe not; a file of code that used (current-location) would usually be semantically equivalent to a file that spelled out the same string literal by hand. In a language with separately compiled modules, both files might compile to the same result, which would make that semantic equivalence precise. In such a language, we typically wouldn't have any reason to mind if a module used (current-location) in its source code, even if we preferred to avoid it for some reason in our own code. This makes it into some kind of "safe" side effect, if it's even a side effect at all.

Nevertheless, within a single file, the expression (current-location) could look the same in two places but give different results.

That's where using `unquote` in table keys becomes quirky: The source code of two table keys may look identical (and hence cause a duplicate key conflict at the source code level) even if the programmer thinks of them as being different because they eventually generate different results.

Because of this quirk, the programmer may have to use some kind of workaround, like putting slightly different useless code into each key:

  '`{,(do 1 (current-location)) "foo" ,(do 2 (current-location)) "bar"}

So yes, we should document this, but not just in this particular case of tables. It feels more like something to bring up in the tutorial.

Do you have particular reasons in mind? It sounds like you're reserving those until you understand what I'm saying with my quasiquoted table examples, but I think those examples are mostly incidental to the point I'm making. (I'll clarify them in a separate reply.)

Maybe I can express this again in a different way.

I bet we can at least agree, on a definitional level, that quotation is good for constructing data out of data that's written directly in the code.

I contend quotation is only very useful when it comes to code generation.

If there were ever some kind of data we could quote that we couldn't use as program syntax, then we could just remove the quotation boundary and we'd have a fresh new design for a program syntax, which would bring us back up to parity between quotation and code generation.

In a Lispy language like Arc, usually it's possible to write a macro that acts as a synonym of `quote` itself. That means the set of things that can be passed to macros must be a superset of the things that can be passed to `quote`. Conversely, since all code should be quotable, the set of things that can be passed to `quote` must be a superset of the things passed to macros, so they're precisely the same set.

No, I think I disagree there, assuming I'm understanding you correctly.

One common case where I used to use quasiquote was in data migrations, and there was never a macro in sight. I don't precisely remember a real use case involving RSS feeds and user data back in the day, but here's a made-up example.

Say you're running a MMORPG that started out in 2D, but you're now adding a third dimension, starting all players off at an elevation of 0m above sea level. Initially your user data is 2-tuples that look like this:

    (lat long)

    (x y z)

Here are two ways to perform this transform. Using quasiquote:

    (whiler (other-user-data ... (lat long) ...)  (read)  eof
      (prn `(,other-user-data ... (,lat 0.0 ,long) ...)))

    (whiler (other-user-data ... (lat long) ...)  (read)  eof
      (prn (list other-user-data ... (list lat 0.0 long) ...)))

That's interesting.... How would you describe what quotation does, then, if you wouldn't say it lets you write certain data directly in the code?

---

In your data migration example, I notice you're reading and writing the data. You're even putting newlines in it, which suggests you might sometimes view the contents of that written data directly. If you're viewing it directly, it makes sense to want the code that generates it to look similar to what it looks like in that representation.

It's not always feasible for code to resemble data, but since that file is plain text with s-expressions, and since the code that generates it is plain text with s-expressions, it is very possible: First you can pretend they're the exact same language, and then you can use `quasiquote` for code generation.

You might not have thought of it in that order, but I think the cases where `quasiquote` fails to be useful are exactly the cases where it's hard to pretend the generated data is in the same language as the code generating it.

---

"I've always thought there's a deep duality between quasiquote and destructuring."

I've always thought it would be more flexible if the first element of the list were a prefix operation, letting us destructure other things like tables and tagged values.

I built the patmac.arc library to do this:

Current link: https://github.com/rocketnia/lathe/blob/master/arc/patmac.ar...

Posterity link: https://github.com/rocketnia/lathe/blob/7127cec31a9e97d27512...

One of the few things I implemented in patmac.arc was a `quasiquote` pattern that resembles Arc destructuring just like you're talking about.

I would agree. At least your examples seem to point to that.

I would suggest you flatten your data and add indexes:

  (= data (obj 1 (obj cont "get eggs")
               2 (obj cont "fix toilet")
               3 (obj cont "Build something that works in arc")  
               4 (obj cont "Research Ordered Associative Arrays")))

  (= todo  '(1 2))
  (= today '(3))
  (= done  '(4))

Trying to go down the road of deeply nested tables or alists will only lead you to pain and suffering (at least in arc).

i.e while using:

  pipe!todo.1!id 

(fetch pipe todo first id)

  $ arc
  ac.rkt:347:43: tablist: unbound identifier
  in: tablist
  location...:
   ac.rkt:347:43
  context...:
   raise-unbound-syntax-error
   for-loop
   [repeats 2 more times]
   finish-bodys
   for-loop
   finish-bodys
   lambda-clause-expander
   loop
   [repeats 66 more times]
   module-begin-k
   expand-module16
   expand-capturing-lifts
   expand-single
   temp74_0
   compile16
   temp68_2
   ...

Un momento; fix incoming.

The general idea behind the fix is that quoted literals need to be treated as data. Arc now has two new functions for this purpose: quoted and unquoted.

The fact that (quote {a 1}) now becomes a hash table is a little strange. I’m not entirely sure that’s correct behavior. It depends whether (car '({a 1})) should yield a hash table. It seems like it should, which is reified in the code now.

EDIT: Ok, I've force-pushed the fixed commit. (Sorry for the force-push.)

  arc> '(pipe "water")
  '(pipe "water")

  arc> "she"
  "she
  
  arc> 23
  23

  arc> {pipe "water"}
  {pipe "water"}

  arc> {pipe "water"}
  '#hash((pipe . "water"))

It sort of went well, but mostly not. :)

Personally, I found I prefer racket's pretty-printing with the horrible hash tables compared to something like {pipe "water" a 1 b 2 c ...} because if you try to evaluate `items` or `profs` you won't have a clue what the data is without pretty-printing.

not bad for precisely 59 minutes.

Even looked at Assocative Containers[1] before I thought, "alist!"

  {todo:({id 1 name "get eggs" done nil} {id 8 name "fix computer" done t})} 

Let me know if that seems sufficient for now.

You're right that Arc still can't read tables written via `write`. That is definitely worth supporting. Here is an example of how it could work: https://github.com/sctb/lumen/blob/55b14ca8aafeaf6b0ca1b636d...

Eh? You definitely can, in Racket. :) The only problem I know of is that when you write a mutable table, you read back an immutable one.

---

"You're right that Arc still can't read tables written via `write`."

Arc's supported this for as long as I can remember. The only problem is that they come back immutable.

Here's Arc 3.2 on Racket 7.0:

  arc> (= foo (fromstring (tostring:write:obj a 1 b 2) (read)))
  #hash((a . 1) (b . 2))
  arc> (= foo!a 3)
  Error: "hash-set!: contract violation\n  expected: (and/c hash? (not/c immutable?))\n  given: '#hash((a . 1) (b . 2))\n  argument position: 1st\n  other arguments...:\n   'a\n   3"

  arc> (= foo (fromstring (tostring:write:obj a 1 b 2) (read)))
  '#hash((a . 1) (b . 2))
  
  arc> (= foo!a 3)
  3
  
  arc> foo
  '#hash((a . 3) (b . 2))

  {'id 3 
   'c {
      'name "james c clarke" 
      'age 23 
      'addr "1724 Cox Ave. NY, NY 90210"
     }
  }

[1] http://arclanguage.org/item?id=20803

  (= tpipe {todo 
              '({id 1 cont "get eggs" done '(nil)}
                {id 23 cont "fix toilet" done '(nil)})
            week '(nil)
            today '({id 83 cont "Build something that works in arc" done '(nil)})

  arc> tpipe!todo.1!id

    (= npipe {todo 
                (list {id 1 cont "get eggs" done '(nil)}
                      {id 23 cont "fix toilet" done '(nil)})
              week '(nil)
              today (list {id 83 cont "Build something that works in arc" done '(nil)})
              done (list {id 44 cont "Research Ordered Associative Arrays" done '(2019 1 21)})})

    (= unqpipe {todo 
                `(,{id 1 cont "get eggs" done '(nil)}
                  ,{id 23 cont "fix toilet" done '(nil)})
                week '(nil)
                today `(,{id 83 cont "Build something that works in arc" done '(nil)})
                done `(,{id 44 cont "Research Ordered Associative Arrays" done '(2019 1 21)})})

    '(foo bar)

    (foo bar)

So in a similar way

    {foo "bar"}

I don't agree. quoting a list is a way to protect the expression from evaluation, in this case because round brackets normally indicate an expression that needs to be called. A table literal {...} doesn't need protection from evaluation as a callable expression as it's just data and like any other data it should evaluate to itself. And, frankly, it would really suck having to protect that data everywhere in my code because someone wants a really nuanced use case to work.

I've been trying to get the tables generated by the personal data link in news to export as JSON for a while now[0]. Part of the problem seems to be related to this - Racket doesn't seem to know what to do with #tagged tem or #hash (much less nils.)

   (defextend writefile (val name) (is (type val) 'table)
     [the body of the save-table function here]

From a bigger-picture point of view, this seems like it would become a non-issue once Arc had its own reader. I assume the problem with reading tables using `read` is that Racket's reader constructs immutable hashes. An Arc-specific reader would naturally construct Arc's mutable tables instead.

Doesn't Racket's reader give us similar problems in that it reads immutable strings and cons cells, too? So these problems could all be approached as a single project.

In the short term, it's not a project that would need a whole new reader. It could just be an adaptation of Racket's existing reader... something like this:

  (define (correcting-arc-read in)
    (let loop ([result (read in)])
      (match result
        
        ; TODO: See if this should construct a Racket mutable cons cell
        ; instead (`mcons`). Right now this just creates an immutable
        ; one, which should be fine since Arc uses an unsafe technique
        ; to mutate those.
        [(cons a b) (cons (loop a) (loop b))]
        
        [ (? hash?)
          (make-hash
            (map
              (match-lambda [(cons k v) (cons (loop k) (loop v))])
              (hash->list result)))]
        [ (? string?)
          ; We construct a new mutable string with the same content as
          ; `result`.
          (substring result 0)]
        ; We handle tagged values, which are represented as mutable
        ; Racket vectors.
        [(? vector?) (list->vector (map loop (vector->list result)))]
        
        ; We handle various atomic values. (TODO: Add more of these
        ; cases until we've accounted for every writable type Arc
        ; supports. Alternatively, just make this a catch-all
        ; `[_ result]`.)
        [(? number?) result]
        [(? symbol?) result])))

Racket also has mutable hashes created using `make-hash`[0] rather than `hash`. It could just be that the tables are not serialised as something Racket reads as mutable hashes when deserialising it back again?

I'm pretty sure the Racket reader never reads a mutable hash, but that it's possible for a custom reader extension to do it.

Some of Racket's approach to module encapsulation depends on syntax objects being deeply immutable. In particular, a module can export a macro that expands to (set! my-private-module-binding 20) but which uses `syntax-protect` so that the client of that macro can't use the `my-private-module-binding` identifier for any other purpose. If the lists constituting a program's syntax were usually mutable, then it would be hard to stop the client from just mutating that expansion to make something like (list my-private-module-binding 20), giving it access to bindings that were meant to be private.

I think this is why Racket's `read-syntax` creates immutable data. As for why `read` does it too, I think it's just a case of `read` being a relatively unused feature in Racket. They don't have many use cases for `read` that they wouldn't rather use `read-syntax` for, so they don't usually have reasons for the behavior of `read` to diverge from the behavior of `read-syntax`.

All this being said, they could pretty easily add built-in syntaxes for mutable hashes, but I think it just hasn't come up. Who ever really wants to read a mutable value? In Racket, where immutable values are well-supported by the core libraries, you aren't gonna need it. In the rare case you want it, it's easy enough to do a deep traversal to build the mutable structure you're looking for (like my example `correcting-arc-read` does).

No..?

    > (require racket/serialize)
    > (define basket (make-hash))
    > (hash-set! basket 'fruit 'apple)
    > (write-to-file (serialize basket) "basket.txt")
    > (define bucket (deserialize (file->value "basket.txt")))
    > (hash-set! bucket 'fruit 'banana)
    > bucket
    '#hash((fruit . banana))

Looks like you're proposing to use a two-stage serialization format. One stage is `read` and `write`, and the other is `serialize` and `deserialize`. At the level of language design, what's the point of designing it this way? (Why does Racket design it this way, anyway?)

I can see not wanting to have cyclic syntax or syntax-with-sharing in the core language just because it's a pain in the neck to parse and another pain in the neck to interpret. Maybe that's reason enough to have a separate `racket/serialize` library.

I've been trying to respond to this, which was your response to that one:

"Racket also has mutable hashes created using `make-hash` rather than `hash`. It could just be that the tables are not serialised as something Racket reads as mutable hashes when deserialising it back again?"

In the `correcting-arc-read` comment, I used `make-hash` in the implementation of `correcting-arc-read`, so I assumed your first sentence was for the edification of others. I found something to respond to in the second, which kinda pattern-matched to a question I had on my mind already ("Can't the Racket reader just construct a mutable table since what was written was a mutable table?").

As for my response to your "No..?"...

One of the purposes of `correcting-arc-read` is that (when it's used as a drop-in replacement for Arc's `read`) it makes `readfile1` return mutable tables. So if anyone had to be convinced that a reader that returned mutable hashes could be implemented at all in Racket, I thought I had shown that already. When it looked like you might be trying to convince me of something I had already shown, I dismissed that idea and thought you were instead trying to clarify what your proposed alternative to `correcting-arc-read` was.

Seems like I've been making bad assumptions and that as a result I've been mostly talking to myself. Sorry about that. :)

Right so, Blockchain is a methodology for a cryptographic system of record with transaction capability, and doesn't have to be used for currency, even though that's how everyone is using it. When you simplify it down to a 'system of record' then other use cases become new opportunities. I don't currently have useful ideas outside of currency, but my point was that Buffet doesn't need to understand the details of the methodology (bottom up) to establish an understanding of Bitcoin for investment purposes.

Even as a currency it is valuable provided that it gets tied to something tangible/meaningful. Personally I'm interested in ethereum because of its smart contracts and dapps.

> Gold has a lot of utility,

I'm aware that gold has some utility, so you are correct that I shouldn't haven't stated 'no utility', but gold's utility is not capable of justifying the price on the market. If gold was not being held as asset for the sake of being held as an asset, based on its rarity, then its market value would plunge to the near bottom.

> I'm skeptical of the notion of intrinsic worth. I think things have value because humans value them.

Right so, Gold has value because humans value it, but its value on the market is not tied to its intrinsic worth it's tied to its rarity alone. And that's the point of highlighting intrinsic worth over a human valuation. The value stability/durability is tied to its intrinsic worth.

The overall point is that Buffet is a value investor that's known to use intrinsic worth as the value measure. This is why Buffet hates both gold and Bitcoin. So let's not assume that he doesn't understand. Bitcoin's value is not stable because it's not tied to anything tangible. Even $ are just notes, but they are backed by a gov't which is why they have intrinsic value and therefore much more stability (or as much stability as the gov't/country provides).