I woke up this morning and discovered that a critical part of our database is down. I am trying to get in touch with our hosting provider to find out if there's anything I can do to recover it. This is ... not a great situation, to say the least, but I'll keep everyone posted.
Hi folks! The server decided to take an early holiday break. Time zones being what they are, our engineer won't be able to look into this for several more hours. I'll keep everyone posted here, on Twitter, and on Tumblr.
Thank you for your patience.
ETA as of 3pm EST: We're back! Go forth and enjoy some fic for your holiday.
Known issue: if you try to edit a chapter, the editing interface will load, but you'll see blank author's notes and story text fields. This is probably related to the fix put into place to address posting new stories, and our engineer will be looking into it. (And note that even though you'll see that empty story box, your content is still safe on our server and has not been deleted by clicking "edit chapter.")
In the meantime, if you need to edit a chapter, please copy the text and paste it into the story box. (Or better yet, make edits in your own document and paste or upload that into the story box.)
The site has been back up for a while (good) but attempting to post stories is yielding a database error (bad, obviously). We're looking into it and will update when we have more information.
ETA: As of 12:30am UTC, posting stories works again. Please contact us at help@whofic.com if you encounter any other issues.
Teaspoon will be migrating to a new server between 1pm UTC on Thursday, September 10 and 2am UTC on Saturday, September 12. (Use this site or any other converter to see UTC time in your local time zone.) During the migration, the server may be unavailable periodically, and to avoid potential data loss, we will not be approving stories until the migration is complete. I'll update this post once the migration is done.
ETA as of 1:30pm Eastern time: the site migration is complete, and the site is loading. DNS information is still propagating throughout the internet, though, so you may find that you aren't able to load it for the next 24-48 hours.
I've reported the outage to our hosting provider, who says it's an issue with the VPS node we're on. This means that it could be a few hours or more before we're back, but I'll update this post when I have more information.
ETA as of 9:46am EDT: We're back! Thank you for your patience.
Unfortunately, we've been down for a solid half hour. I've been in touch with the hosting provider, and their support team should be looking into this soon. I'll post more information as soon as I have it, and apologies for the unexpected downtime.
9:24pm EST: We're back! Our hosting provider confirmed that there was hardware failure on their end, but everything is fixed now.
This weekend, we're upgrading our version of MySQL. While we don't anticipate any significant issues with the upgrade, please be advised that some downtime could occur.
During the weekend of December 28, we're going to switch Teaspoon's skin over to the beta skin we released earlier this year. We don't anticipate any downtime as part of this switchover, but obviously, will keep you all updated in the unlikely event something goes wrong.
What this means for you if you're still using the current "bluepurple" skin:
We will be removing this skin entirely. It isn't responsive – that is, it doesn't scale smoothly from desktop monitor size down to phone screens – and we can't address lingering display issues in the beta "neospoon" skin until bluepurple is removed.
That said, we have a responsive version of bluepurple ready to go! So if you prefer the current design to the new one, you'll still be able to switch to something that looks very similar to it. (It's got the same layout as neospoon, but uses bluepurple's color scheme and familiar crossed spoon/sonic screwdriver background.)
We are temporarily down. It’s an issue on our hosting provider’s end and affects other servers as well, so there’s unfortunately nothing we can do at the moment to fix things. We’ll be back as soon as we can.
It is way, way past time for us to do this, but we finally have a mobile-friendly design in beta! I could post a whole pile of excuses and explanations about why this hasn't happened until now, or I could just point you to what you care about: more information about the beta skin and how to enable it.
If you're like every other human being who uses the internet, you've probably gotten a bunch of emails recently from sites talking about their privacy policies and the EU's new GDPR regulations. To the best of our knowledge, Teaspoon is not subject to the GDPR, but it's still in everyone's best interests to make our privacy policy as transparent as possible.
This policy is now available on the site as part of our Terms of Service page. None of the policy is new -- we have never required any personally identifying information from you other than an email address (which can also be anonymous!), and we don't share your data with anyone outside the moderator team unless you choose to make it available yourself. But we haven't logged the policy publicly until now.
If you have any questions about this, feel free to contact us at help@whofic.com, or leave a comment on this post.
Update as of January 2, 6:25 EST: WE ARE BACK. If you don't see us, wait a little while and try again; the DNS update may not have made it your way yet.
Our hosting provider recently performed some routine maintenance on the server cluster we share with other sites, and as part of that maintenance, we were just informed (and I do mean "just" -- the mail arrived on Christmas Day) that they are issuing new IP addresses that require us to change part of our site configuration. We're in the middle of that change now, which may result in the site being temporarily unavailable or looking unavailable even though it isn't. We expect this to resolve itself automatically within the next 24-72 hours, if not sooner. (We unfortunately cannot control the speed at which this problem resolves itself, because it has to do with DNS propagation worldwide.)
ETA as of December 31, 2:30pm EST: the site is down for unknown reasons. We're hoping it's due to the IP address switchover, but are having difficulty reaching our hosting provider, given the holiday weekend. We'll be back up as soon as we can.
If you've noticed that you can't get to Teaspoon right now, that's because our hosting provider tells me there's a networking issue with the virtual private server node we share. It's being worked on, and they hope to have it resolved soon.
With the premiere of the new Doctor Who spinoff show, Class, scheduled for this weekend, we've added a category and characters for it to the site. Post your stories anytime!
ETA: We believe our hosting provider has resolved the problem, which was due to security restrictions set as part of their nightly maintenance. If you are still running into this problem, however, let us know.
We've started to hear from some users that they're getting 403 errors ("Access denied") when trying to load any page on the site. Since we haven't changed anything about our site configuration, the problem is either at our hosting provider or somewhere upstream; we're working with our hosting provider now to try to narrow down the source of the problem.
What I suspect, unfortunately, is that there is some higher-level blocking and/or interference going on from AT&T, because so far, they are the one common factor to all the reports for which I have sufficient data. I'd like to collect more data to either support or disprove this theory so that the hosting provider and our mod team can figure out what to do next. (Which I genuinely hope is simpler than contacting AT&T and telling them to knock it off, because I have grave doubts about that being a timely or effective solution.)
If you are having this problem, please reply to this post with the following:
What device(s), operating system, and browser you're using when you get the 403 error. Please also include the device manufacturer; so far, we have one report of an LG tablet not being able to access the site, but most reports come from iPhone users.
Whether the problem is happening on cellular data, wifi/wired connection, or both
If the problem only happens on your data plan, the name of your mobile provider
Your IP address when the problem occurred -- you can get this from iplocation.net
Thank you, and if you're encountering this problem, thank you for your patience. I wish I could say this would be resolved quickly, but because it isn't a problem we created (or can fix ourselves!), I honestly don't know how long this will take. However, I will post an update when I have a better idea of what's going on.
The hosting provider is still working on it. I have no further update from them beyond 12:43pm EDT, the point at which they said they were going to try a more recent backup. (We know the September 19 backup works perfectly, and worst case, they will fall back to that and export the database from a day or two ago so that we will have lost virtually no site data.) While I am also still waiting on a detailed explanation about what caused the crash and how or if we can prevent it from happening in the future, to the best of my knowledge, the issue is entirely due to automated operating system upgrades gone very wrong.
This is a frustrating situation to say the least, but I will continue to keep people updated here and on my Tumblr. To avoid spamming your flists with posts from this comm, any further updates will be added to this post rather than a new one.
UPDATE AS OF 7:55PM EDT: We're back, but on the September 26th backup. If you posted your story or comments between 2:26am Central Daylight Time and 5:52pm September 30 CDT, we no longer have it. I'm very sorry there's been data loss, but please resubmit any missing stories or chapters, and we'll reapprove them as soon as we can.
PLEASE DO NOT SUBMIT STORIES OR COMMENTS TO THE SITE RIGHT NOW. With any luck, it will be going back down shortly so that this can't happen in the first place. Yes, I know there are no recent stories listed, but the stories are still there.
(I have to run out the door to go teach now, so cannot provide a detailed explanation at the moment -- but the short version is that if you submit stuff now, it may be lost.)
I don't think it will be for more than a few hours, but there were some issues with the server software (not a hack attack, not the eFiction software, literally a non-security-related problem with the OS) that have taken us down. Our hosting provider is on the case and working to restore things now.
I apologize for the outage. We'll be back as soon as we can.
Our help address is working again, so feel free to contact us at help@whofic.com (the address in Teaspoon's footer) if you need assistance.
Also, calufrax, Teaspoon's recommendations community, is looking for reccers. All you need to do is select 4-10 stories that haven't been recced before and that meet the comm's fairly basic rules (well-written/formatted, no bashfic, etc.). Reccing periods last two weeks, and your recommendations display on Teaspoon's home page! To volunteer, see the complete rules and sign-up instructions in this post.
For some reason, our help email forwarding isn't working. We're looking into the problem, but if you've sent us a message since July 19 and haven't heard back from us, that's why. For now, please contact us at whofic on Gmail.
Hopefully this will be a quick fix. I'll post an update when I have one.
The line-break bug affecting submissions to the queue is fixed, so we're going to make our way through the queue now. Feel free to start submitting more chapters and stories!
We're up! Depending on how long it takes your DNS to update, you may have a problem getting to whofic.com over the next 24 hours, but that will fix itself on its own. I cannot thank our engineer, abates, enough for his help here, or our hosting provider, HostForWeb, who really came through with that backup and reinstall.
Our next steps are to investigate an upgrade to eFiction 3.5.5. I'll post more info about that when I have it.
Please note: as I suspected, some authors did submit chapters to the queue after yesterday's backup was completed, so those chapters did not make it into the backup. If you submitted your chapter after May 29, 2:16pm Central time, it's gone; please resubmit, and we'll look at it in the order in which it's received.
ETA a new and important note: I have just discovered that it seems like we’re missing a customization that allows line breaks to be processed, so everything I’m seeing in the queue looks like a giant wall o’ text. It will not be possible to fix this before our engineer is available again later today, so you may want to hold off on submitting any new chapters until you hear from me. No chapters currently in the queue will be approved until this problem has been fixed.
Previous