trinque: jfw: there are patches written by a guy named funkenstein which I've used
trinque: more or less a backport of sendrawtransaction
trinque: I also have a patch on my desk for eradicating the entire wallet
trinque: I did not publish because at the time the propaganda read "reference implementation" and without the wallet, it wouldn't have been.
jfw: oh hi trinque. I had found funkenstein patches for importprivkey + dumpprivkey and removing checkpoints but not raw tx, do you have a link for that?
jfw: oh look at that, http://btcbase.org/patches/polarbeard_add_sendrawtransaction_rpc
trinque: ah that was it, had the wrong guy.

The patch will need some scrutiny but isn't huge.

"I'm bias to say it makes sense to make a new article for clarity's sake." - Done and let me know if it satisfies.

[re bignum] "what remains for it to be ready for v1 ?" - it works; just might take a couple minutes of CPU to sign transactions. Still beats doing 256-bit division by hand, right??

"how's this version read to you ?" - Stylin'!

"reports as the tasks are completed." - Will IRC be acceptable to you for this given the now relatively detailed task breakdown?

"I think the pain is tolerable for now." - I agree, if it turns out necessary

cool.

"I think implementing the less fragile -- pushrawtransaction rpc -- as the higher priority. I take from your description that pushrawstransaction is also the simpler implementation, is that correct ?" - I think it would be the simpler implementation overall, though noting that pushrawtransaction would be a TRB patch whereas peer connection would be independent. I intend to ask trinque about this too: perhaps there's an existing solution by now.

Ok, report back here what trinque says.

At present I think TRB vpatch remains the way to go. One should be able to independently push a transaction.

"Not saying we should change the schedule, pointing it out we can make it work if it's worth it." - ok. Another opportunity cost of extending the schedule now could be the OS project, as I'm sure you're weighing.

Right, that changed the present situation.

"The spec article will : a) provide a basis to evaluate the difference between the planned functionality and usage for v1 and the code you currently have ;" - hmm, I thought this was done in what I already wrote. Could you clarify what you see as missing there?

My reply there : please clarify, what, if any, revisions are needed. I'm bias to say it makes sense to make a new article for clarity's sake.

"b) provide a basis for QA to work from and for user documentation to be written" - maybe this is some old folly on my part but... how does one test or document something that doesn't yet exist? These sorts of documents cover details that are not yet certain; and it's not like these tasks are going to be done in parallel by someone other than me. (Right?)

I know there are some details to be worked out, my thinking is write the spec as far as you know it today and update it as it clarifies. I need to stay with you in parallel and you need to describe the status and choices when you run into a business decision. Seems to me the spec is a central document to maintain coherence for both ourselves and who ever else wants to contribute.

I appreciate the effort to get the plans crystal clear; though I wonder if it's a luxurious level of consideration that we can't afford right now. I do at least owe an update on tasks and time estimates and perhaps this will clarify.

Things have changed since we started working on this, the update on tasks and time estimates will certainly clarify. I look forward to it.

"Bignum change" - unfortunately I found it is not a quick job to switch to the new builtin operators. I wrote those to the Scheme standard; like Ada, this omits bit shifts, and my extensions for that only covered fixnums. There are more details included in the old bignum code that would need porting for the new builtins, though shifting looks like the major one at first glance.

Using those fixnum extensions I was able to replace some quotient/remainder ops in the older bignum code with bitwise ops, which gave a 12% speedup in address generation. Further optimizations are possible but would require a deeper dig from what I'm seeing.

One (unsurprising) lesson here is that when performance counts, language support needs to map well to what the hardware actually does.

Is this in a stable state for v1, or what is the next steps there/what remains for it to be ready for v1 ?

"Is it agreed then v1 will be S-expression and not JSON ?" - Yep.

"So sqlite on online side and plain text on offline side, right ?" - Right.

Aye.

"No one, clients included, wants to see Jacob kill himself to deliver on an unnecessary deadline when a couple weeks extension allows actual sustainable growth." - Thank you; pretty sure the extra month is not needed to buy my life, as long as the plan is able to proceed!

Cool, as long as we know what the plan is and correctly maintain expectations for delivery with them.

You can use [blockquote] in comments btw; might be simpler than all those >'s.

how's this version read to you ?

"What is the current plan for documentation ?" - I plan to write user-level documentation; this should at minimum be included as a text file with the code, so you can read offline on the wallet machine. I don't see a source code level dissection being at all required for this delivery, do you? Nice to have once the code is well stabilized. Can we make that decision once we get there?

Ok, so I'll have a plan with time estimates/due dates to read and reports as the tasks are completed.

Then a source code level dissection as part of v2 when we get there.

"Do you agree those are priorities ?" - should be covered by the above.

So source code level dissection article series is off in this case, and spec is either original article with updates where revisions needed or a new article with revisions included.

"I think implementing the less fragile -- pushrawtransaction rpc -- as the higher priority. I take from your description that pushrawstransaction is also the simpler implementation, is that correct ?" - I think it would be the simpler implementation overall, though noting that pushrawtransaction would be a TRB patch whereas peer connection would be independent. I intend to ask trinque about this too: perhaps there's an existing solution by now.

"Not saying we should change the schedule, pointing it out we can make it work if it's worth it." - ok. Another opportunity cost of extending the schedule now could be the OS project, as I'm sure you're weighing.

"The spec article will : a) provide a basis to evaluate the difference between the planned functionality and usage for v1 and the code you currently have ;" - hmm, I thought this was done in what I already wrote. Could you clarify what you see as missing there?

"b) provide a basis for QA to work from and for user documentation to be written" - maybe this is some old folly on my part but... how does one test or document something that doesn't yet exist? These sorts of documents cover details that are not yet certain; and it's not like these tasks are going to be done in parallel by someone other than me. (Right?)

"c) help you refine the question for MP and prepared yourself with coherent context to link him to." - likewise, do you see incoherence in the last article? The main missing point I see is the data structure detail, which is what the question concerns.

I appreciate the effort to get the plans crystal clear; though I wonder if it's a luxurious level of consideration that we can't afford right now. I do at least owe an update on tasks and time estimates and perhaps this will clarify.

"Bignum change" - unfortunately I found it is not a quick job to switch to the new builtin operators. I wrote those to the Scheme standard; like Ada, this omits bit shifts, and my extensions for that only covered fixnums. There are more details included in the old bignum code that would need porting for the new builtins, though shifting looks like the major one at first glance.

Using those fixnum extensions I was able to replace some quotient/remainder ops in the older bignum code with bitwise ops, which gave a 12% speedup in address generation. Further optimizations are possible but would require a deeper dig from what I'm seeing.

One (unsurprising) lesson here is that when performance counts, language support needs to map well to what the hardware actually does.

"Is it agreed then v1 will be S-expression and not JSON ?" - Yep.

"So sqlite on online side and plain text on offline side, right ?" - Right.

"No one, clients included, wants to see Jacob kill himself to deliver on an unnecessary deadline when a couple weeks extension allows actual sustainable growth." - Thank you; pretty sure the extra month is not needed to buy my life, as long as the plan is able to proceed!

You can use <blockquote> in comments btw; might be simpler than all those >'s.

"What is the current plan for documentation ?" - I plan to write user-level documentation; this should at minimum be included as a text file with the code, so you can read offline on the wallet machine. I don't see a source code level dissection being at all required for this delivery, do you? Nice to have once the code is well stabilized. Can we make that decision once we get there?

"Do you agree those are priorities ?" - should be covered by the above.

> diana_coman: jfw: but as far as I understood the situation now, the focus was on getting a *working* version that keeps to republican fundamentals and that hopefully you can further iterate upon as you gain experience with your biz.
> jfw: that's right
>

Agreed.

> diana_coman: fwiw, I'm also not that sure that dorion's point was re getting it perfect now or something, eg "sufficiently extensive talks upfront and relatively less time coding than a brief terse exachange and longer than expected coding time." - this reads to me more like making sure it's clear what is to implement and how long it will take ie clarifying not perfecting.
> diana_coman: then again, you talked to him, not me.
> diana_coman: jfw: put it also the other way: if you had this one extra month, what do you think would be delivered additionally?
> diana_coman: give him the options: this month like this or in January like that.
> jfw: I answered the "sufficiently extensive" point but he didn't quite clarify that I saw; perhaps I'm indeed over-interpreting.
> diana_coman: sigh; /me goes to read your talk there.
>

My principal aim in sufficiently extensive talks is to clarify. It's been a long time since this wallet project was under development and I want to make sure I understand where it is and where it's going when. Was for sure, "If we have another month, what does that buy us?" Note: I don't ask that only from a code point of view. No one, clients included, wants to see Jacob kill himself to deliver on an unnecessary deadline when a couple weeks extension allows actual sustainable growth.

> jfw: what would be delivered additionally is something that's perhaps received more input from the wider republic - although it's possible that such feedback can't be accomodated by January, e.g. if it's to rewrite in Ada or something
> diana_coman: jfw: that's not something feasible on that scale by end of January, no; and do NOT even think of re-writing it now because you can get an extension, GAH.
> diana_coman: jfw: he said correctly " What gaps remain ? What questions arise now in reading these articles ? Let's go ahead and leave our questions as comments on those articles." - that's about as much additional feedback you can get and then see if it squeezes through in this implementation; so: any questions there?

> diana_coman: jfw: tbh I think he stated there quite clearly for each point what next so not sure what is left other than the above ie if there are further gaps and/or questions on the fundamentals there
> jfw: fair enough; I didn't see any big questions on the fundamentals, more on the details. Can check if there's anything that can be asked in the context of the articles.
>
> diana_coman: and this "evaluate the usage within the context of the skills they either already had or are building through working with us" - so how well does this v1 fit what your clients learnt to use? to what extent would pushing the deadline to January increase this fit?
> jfw: I don't see it increasing the fit, they'd be learning something new building on existing skills either way.
>
> diana_coman: hm, I thought you two were better at talking to one another, huh.
> jfw: we or at least I might be struggling a bit with the more async form
>

One thing to consider is, with extra time, how much better can/will the documentation be ? What is the current plan for documentation ? yrc and Gales Linux come with well written plain text documentation. You didn't have a blog when you published those. Now that you have a blog, does an FFA/Eucrypt style article series of the vpathes make the most sense ?

> diana_coman: so then tell him all that ie 1. fundamentally it's aligned and fine, will check details today/tomorrow and see if there are further clarifications to ask for at those articles 2. not much to gain in extending the deadline as v1 of the wallet as it is would be a reasonable fit and either way they would still be learning something new (though ahem, do note that there are *degrees* there ie how much of a leap)
> diana_coman: 3. can iterate in principle more usefully once this version is out and in use (if that's the idea).
> diana_coman: jfw: I think I covered there main points but see if there's anything else you had in mind.
> jfw: thank you diana_coman; I will review the points.
> diana_coman: jfw: if you have something more concrete re Jan delivery, add that eg if we push the deadline to January, we can also have this and that that might/might not make it slightly/not at all/a lot easier for clients to learn/use/whatever.
> diana_coman: jfw: and you know, reply to his points there eg will do etc.
> jfw: diana_coman: will review & ack *all* the points then, right.

Thanks for covering all the points, I think I covered all yours here.

At this point I see the next steps being an article to specify wallet functionality and usage and deciding if article series publication style is the best distribution strategy. Do you agree those are priorities ?

The async comms were difficult this week, but I think it's been productive. Back near stable power and connection so comms to normalize.

> Diana Coman's pointed out some holes in my communication, so I'll make a fresh pass through the whole thing and then import the log.
>

Thanks, this was helpful.

> About emailing them - if we do this, we ought to work out between us a more specific choice to propose i.e. *this* is what can be gained by extending the deadline. But on that, I'm not seeing much to be gained in usability in context of their skills. Getting an initial version out should provide more clarity from both the technical and business view on what if any additions would be desired in a v2. Any sort of rewrite is certainly not going to fit in January (not to imply you suggested it would).
>

Agreed on the communication with them.

Agreed that the foundations of what you have are within their skill set now and agree on getting the initial version out and no rewrite was implied really.

> On the articles and gaps - the present code and remaining plan is fundamentally in accord with them as far as I can see, with the exception of advanced accounting and optimal input selection, or other things that might be implicit under "better support code and integration generally". On input selection by ordering of a text file, I'm not certain yet how this works in practice because I'd been thinking of the data as a chronological list of transactions rather than reorderable list of unspent outputs. The latter could be extracted from the former on demand. Perhaps this is something that I could refine into a question for MP. In the categories of the "cutting the wallet" article, the design is most closely described by II, the solipsistic node: the offline part knows only about the transactions that affect it, with new outputs ("coinbases" there) relayed manually, thus is light on data storage yet has what it needs to create transactions.

Right, that confirms my read. I think the next step for v1 should be for you the write the specification for the wallet and publish it as an article here.

The spec article will : a) provide a basis to evaluate the difference between the planned functionality and usage for v1 and the code you currently have ; b) provide a basis for QA to work from and for user documentation to be written and c) help you refine the question for MP and prepared yourself with coherent context to link him to.

>
> Bignum change - I'll take that hour tomorrow.
>

Nice, how'd that turn out ?

> "On import to update state, append to text file encoded transaction data is fine for now." - OK (note that this is based on the "chronological list of transactions" format).
>

Ok and noted.

> "JSON interoperability" - per comment 4.

Is it agreed then v1 will be S-expression and not JSON ?

>
> "On pushing raw transactions to trb" - per comment 4.
>
> "Let operator set a cron job if he finds himself cranking more than he wants." - you'd want to avoid a cron job rewriting text files while you're accessing them elsewhere; also I realized that the online component needs to track more state than previously mentioned, specifically the unspent outputs so that it knows when they're spent. Together these seem to me to argue in favor of a proper database, both regarding robustness and time, which I don't see being in conflict with the principle of plain-text for the offline part. As you know I'm pretty familiar with python/sqlite which is already in our stack.

That's a good point about the cron job, hm.

So sqlite on online side and plain text on offline side, right ?

> 4. Yes, though changing storage formats could be a pain for users.

I think the pain is tolerable for now.

> 9. Either a pushrawtransaction or connecting on port 8333, doing the version handshake and such to relay the transaction as if from another node. However the latter's more fragile e.g. if the node is running -nolisten or all inbound slots are in use, and the former's preferable per http://fr.anco.is/2017/01/removing-the-wallet-from-trb-first-thoughts#source_1 .

I think implementing the less fragile -- pushrawtransaction rpc -- as the higher priority. I take from your description that pushrawstransaction is also the simpler implementation, is that correct ?

> diana_coman: tbh I'd much rather incline at this stage to *not* change the plans now but you do need to talk to him and agree if that's the best way forward atm or not.
> jfw: aha, was my inclination too
> diana_coman: is it perhaps a problem to have next year some "version 2" ?
> jfw: no; though I think his concern is trying to get the plan right the first time so as not to have to redo things
> diana_coman: eh, that he won't ~ever get, lol.
> jfw: heh, and the v1 would be instructive in further planning I expect
> diana_coman: quite.
> diana_coman: it's more important to not get it catastrophically/fundamentally wrong, that's about it.
> diana_coman: that said, there IS a cost to having clients change versions though
> diana_coman: hence my question above whether this is a potential problem or to what extent

My main purpose in pointing out the potential flexibility is to identify it as an option, such that we don't box ourselves in from an external deadline that's not actually there. If we actually have a little extra time, to what extent is making use of it worth it ? Not saying we should change the schedule, pointing it out we can make it work if it's worth it.

> jfw: I'll add that to the list to clarify with them.
> diana_coman: jfw: that is more to clarify with dorion really; they won't *know* to tell you exactly, how could they?
> diana_coman: sure, you'll get an answer if you ask but ...

Yeah, let's come to a decision and then tell them rather than ask.

> jfw: On reviewing the trilema wallet articles he linked, it looks like I've covered most of the points, except for things like better accounting functions and programmable input selection, which I'd think can be added later once better specified
> diana_coman: so then it sounds like best to get v1 working really.

Agreed to both.