MULTIMAX Internet Downtime - Explanation

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
24/03/2012 12:13

Hello All,

At approx 10:20am this morning, our Internet has went down unexpectedly from an problem that was between our Network Operations Center (NOC) and the Turkish Telekom.



All our systems were working within normal parameters and there was no equipment down on our side, but our Internet connection was not working. We have immediately contacted technical department of TTNet in Turkey as well as officials here in Local Telekom Office, in the meantime, we have sent a team down to our NOC to assess whether there is a physical malfunction with the fiberoptic cable between us and the telekom office.



I must say that the Local Telekom representative, Mr. Engin was very helpful, he immediately took off from his house to the Telekom office even though it was in his off day and was on site within 20mins.



After speaking with Turkish Telekom in Turkey they have reset our port and the internet was back up, without any physical interference from our side.



(cont)

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
24/03/2012 12:14

Total downtime was approx 55 minutes, all our customers will be receiving 1 day extension on their accounts as per day-for-hour compensation scheme.



I apologize for this inconvenience and thank you for your understanding.



Kemal

Supersnooper


Joined: 27/09/2011
Posts: 295

Message Posted:
24/03/2012 12:22

Kemal, thank you for the explanation. Wow, what excellent service from you. Soooo glad I changed provder!!

loulou



Joined: 14/05/2008
Posts: 785

Message Posted:
24/03/2012 12:35

Excellent service my friend

ruggy


Joined: 02/05/2009
Posts: 757

Message Posted:
24/03/2012 13:03

Kemal, I have just paid for 12 months 4 meg, get 15 months, but i was not given any paper work in relation to that, what should i do. Great service, well done.

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
24/03/2012 13:39

@ruggy, I have checked your account and it is indeed extended for 15 months, you can pick up a copy of your original invoice from out Girne office. Thanks.



K.

Simhar


Joined: 18/05/2009
Posts: 227

Message Posted:
24/03/2012 14:16

Great customer service.... again..



Thanks Kemal

herindoors


Joined: 13/01/2007
Posts: 138

Message Posted:
24/03/2012 15:46

excellent customer service as usual, thank you

flutterby


Joined: 11/01/2008
Posts: 214

Message Posted:
25/03/2012 07:07

Have been extremely pleased with Multimax. However, down this morning at 7.00 am. Is this same problem as yesterday, or have I a problem with my internals !!!! Anyone else down??

tomsteel


Joined: 22/06/2009
Posts: 482

Message Posted:
25/03/2012 08:24

Bahceli down as well. Using another provider to send this.

herindoors


Joined: 13/01/2007
Posts: 138

Message Posted:
25/03/2012 09:01

Esentepe down as well this morning 9.am:(

GinaC


Joined: 26/11/2010
Posts: 372

Message Posted:
25/03/2012 09:26

Down inLapta also we have been told its a Turk Telecom issue.

Seasider


Joined: 16/09/2008
Posts: 69

Message Posted:
25/03/2012 11:29

Also down in Catalkoy on ADSL but now up and running. Also Kemal your advert is a bit misleading as the extra 3 months on annual subscriptions does not apply to ADSL- this is not stated in the ad. Otherwise great service particularly from Seran in Girne office and Euli in Lefkosa office.

ttoli


Joined: 24/03/2007
Posts: 1172

Message Posted:
25/03/2012 12:09

Down in Dogankoy too,250 m from the Karakum base station, box on roof showing 5 lights but only on local connection.

techtrader


Joined: 06/08/2008
Posts: 126

Message Posted:
25/03/2012 12:11

back now in Catalkoy (from Cratos) but off all night

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
25/03/2012 12:24

well well well,



words does not explain the mood we are in at the moment.



Our internet went off again the same way as yesterday at around 1am this morning. It took us approx 1 hour to get a hold of somebody on the other side of the line at Turk Telekom in Turkey just to report our problem, hours more to get a feedback on whats going on.



By around 4am, we had eliminated all non physical elements as cause of the problem and concluded that there is something physically wrong with the fiberoptic cable from our NOC to Telekom office.



In the next 2 hours, we had done and repeated all the tests on our side and by 6am, concluded that the fiber on our side is ok.



At that point we had nothing more to do but to wait for the morning and reach a Telekom official to continue work on telekoms side. We have managed this around 10am this morning and after some struggle we have managed to bring the link back up.



(cont)

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
25/03/2012 12:35

However, the problem is still not resolved. We have asked to be put onto another 'better' pair of fiberptic cables (we have 24 pairs coming to our NOC), but the people in charge of that were not available as it was a sunday.



3 of them were called up for help and none of them wanted to come and solve this problem permanently.



We are currently running on fiber pair with low signal, we are hoping that the line will hold until tomorrow morning so we can go and sort out the problem permanently.



It is very frustrating in these times as whatever we do to provide a quality of service, it looks like something happens that we cannot control and throws all our efforts out to the bin.



I apologize to all customers who has tried to reach us and couldnt, but as you can imagine, given the magnitude of these issues, we were definitely not equipped to handle this kind of phone traffic while at the same time trying to coordinate and fix the problem as quickly as possible.



K.

DesiH


Joined: 24/03/2009
Posts: 152

Message Posted:
25/03/2012 12:47

Many thanks for the explanation and for doing your best to get things running again.

herindoors


Joined: 13/01/2007
Posts: 138

Message Posted:
25/03/2012 12:49

Thank you for your explanation Kemal.

elko2



Joined: 24/07/2007
Posts: 4400

Message Posted:
25/03/2012 12:57

I never noticed anything and I am very pleased with the service. Now my wife can watch her normal tv serials and I can watch any channel on my computer via internet with my head phones on.



Just one question to Kemal out of curiosity: When I connect to Google, Google Cyprus comes up. Hence I presumed that we are connected via the dark side but apparently not. What is the explanation for Google Cyprus?

ismet

Hippo


Joined: 02/02/2007
Posts: 2070

Message Posted:
25/03/2012 13:33

Dark forces are working, or is it to much shopping on the Greek side?

techtrader


Joined: 06/08/2008
Posts: 126

Message Posted:
25/03/2012 13:44

noticed same with betfair going all Greek !!........what's the story with Malpas basestation now Kemal is that off the table or might it happen in the future??

erolz


Joined: 17/11/2008
Posts: 3456

Message Posted:
25/03/2012 14:18

Elko2 I suspect Kemal is sleeping, at least I hope he is, as he was up all last night and well into today trying to get the problems detailed above sorted so I'll try and answer you. Google is I suspect picking up 'location' based on the registration details of the IP address range multimax uses, not based on the physical route traffic takes and hence the discrepancy. The IP address block multimax uses for its customers is registered as RoC, but traffic does not physicaly travel via the RoC but like all other internet traffic from the TRNC (bar 2 way satelite systems) via Turk Telecom and Turkey and from there out to the wider internet.

PhilUK


Joined: 31/03/2010
Posts: 236

Message Posted:
25/03/2012 17:57

Minor point Kemal -In the past when multimax has been down I've had a text message telling me what was going on(It also means I'm not thinking there is some problem with my computer )which is great customer service-and It prevents multimax's switch board from being swamped.

However- this morning no Internet,and no text message-so I assumed (wrongly as it turned out!) that my computer was at fault and spent an hour trying to figure out what the problem was.

I think the text message is/was a great idea- but it does need to be consistent.

Otherwise I have been delighted overall with the multimax package.

carian


Joined: 13/03/2009
Posts: 336

Message Posted:
25/03/2012 18:05

Phil, as Kemal states that it went off at 1am this morning, I don't think anyone would appreciate a text at that time. Later on, I'm sure he must have been really busy trying to sort the problem out, plus with it being Sunday there probably aren't as many staff around to help out.



Know its difficult when you're used to the text messages, but at least you do get them most of the time, unlike Nethouse whose customer sevice is non-existent !!

techtrader


Joined: 06/08/2008
Posts: 126

Message Posted:
25/03/2012 18:28

I vote for text anytime!



I was messing around with my tech well past 1am!!

ttoli


Joined: 24/03/2007
Posts: 1172

Message Posted:
25/03/2012 18:31

You can bet you bottom Dollar, that If Texts were sent out in the early hours, that there'd be no end of complaints, guess He cant win either way.

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
25/03/2012 18:34

It is good to wake up and see all questions ask and they are already anwered . As erolz stated we were up all night until the problem was fixed this morning, so as soon as it was, it was bed time for us .



Just to 'officialize' the answers:



1. We are NOT getting any Internet from Greek ('dark') side, not only it is illegal, it is also economically unfeasible as wholesale Internet is much cheaper on this side. The reason why google.com.cy shows up when you use the internet as the default browser is only because when we registered our company with the authority figures in Europe, we have chosen Cyprus as our country as opposed to Turkey, which would be another option. Being 'Geo-IP-located' in Cyprus rather than Turkey presents with some advantages especially as long as Turkey's stand in cencorship goes.



(cont)

PhilUK


Joined: 31/03/2010
Posts: 236

Message Posted:
25/03/2012 18:44

carian msg 25-I get unsolicited texts to my phone 24/7,so I switch my phone off when I'm asleep.

But I didn't discover multimax was off until in the morning I.e around 10am.

A quick text to all your customers cant take long.

As It happens when I realised the problem wasn't at my end I tried ringing Multimax a total of 3 times before giving up-obviously everyone was ringing them.

I still think the text message is the best and most efficient method of keeping customers informed, and agree that they could be sent out between the hours of say, 7am to 11pm, so as not to upset anyone that sleeps with their phone on by their bed.

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
25/03/2012 18:45

2. As for the text messages goes, the reason why text messages did not go out to customers about this problem was because of a couple of reasons:

i) The SMS system is not yet automated, so somebody would have to do it manually, being saturday night/sunday morning, we did not have any employee on duty who would be able to send messages, while we work on the actual problem.

ii) Once identified the problem by 6am, we were optimistic that the problem could be sorted before or around 9am, so concentrated our efforts on that rather than trying to let everyone know that there is a problem. This unfortunately didnt happen as when it was 8:45am yesterday's time that we were able to reach Telekom's officials, it was 9:45am today's time because of the time change. Having already lost 1 hour on time change and the problem taking a bit longer than anticipated to resolve didnt help with our plans.



K.

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
25/03/2012 18:51

@PhilUK, I actually agree with you 100%, a system that updates SMS client database from our actual Customer Database on a regular basis so that we can send SMS'es quickly in situations like this is now our one of the top priorities.



When there is a problem with a specific basestation, it is much easier as each one usually has less than 100 customers, so the sms list can be created and used rather quickly, we just could not be done that for all the customers.



K.

BizziLizzi


Joined: 02/08/2011
Posts: 855

Message Posted:
25/03/2012 19:18

I have been lucky (for once) either my area wasnt affected or I wasnt using the Internet at the relevant times.



However (whilst I appreciate the problems of time in this particular instance) I would like to support the principle of text messages were possible . As a technophobe I tend to go into panic mode when I lose the Internet and knowing that it is not my computer at fault and there is nothing I should be doing really helps a lot!!!

keithcaley



Joined: 13/06/2008
Posts: 2521

Message Posted:
25/03/2012 20:31

Kemal,

I did notice a difference between the most recent problem and the previous one: -

The previous time, I was able to 'ping mmcyprus.com & to reach your homepage with my browser - I actually used the online chat facility to contact a customer service rep, who obligingly updated me.

This time, however, I couldn't even ping your site, let alone display it - it just 'felt' like a different problem - of course it may be that you had to take the servers down while troubleshooting

Anyway - and I did suggest this to 'another service provider' with no response - If your local server is up & running, but you've 'lost contact' with the rest of the world, why not use the server to display a message explaining the problem, a small selection of messages would cover most eventualities.

Would it be feasible to extend this to local hubs, in the event that they lost contact with the rest of the system?

ALSO (I've nearly finished!) pre-recorded messages on your Ansafone would help!

jamestalbot


Joined: 20/12/2009
Posts: 958

Message Posted:
25/03/2012 20:35

Kemal I like message 5 have just paid for the 12 months I payed it at Techmar, I was given a receipt but no paperwork to explain when it becomes due again or any conditions etc.





Regards James

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
25/03/2012 21:50

@Keith, The problem, this time was that because the internet was completely gone, our dns servers could not update themselves and when the cache entries were expired, no website name was being resolved. mmcyp.com domain dns entries are stored on our servers in the UK and therefore the name was not available through dns even though the webserver was up and running, if you did type the ip address http://31.209.96.30 on your browser instead of mmcyp.com it would work and you would be able to use online chat facility.



In regards to redirecting all customers to a warning page, we have actually spent around 15 minutes on this and found out that it does not work as intended without DNS servers working as dns'es needs to resolve the ip first for that IP to be rerouted to another page that we like to show. Interesting fact that we have found out this morning.



(cont)

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
25/03/2012 21:59

Obviously these, together with sms warning systems are shortcomings of our current system.



We were so focused on providing a reliable service so that this does not happen that we have neglected the procedures in the case that it does.



Tomorrow, once the problem is permanently sorted when we move to a proper fiberoptic pair, we will be sitting down and evaluating the situation and make all the necessary precautions.



The agenda so far includes, in priority order:

1. Setting up an auto updating sms system so that it can be used as a one click sms info to affected customers.



2. Setting up a local webpage, where it remains accessible even if there are problems with the Internet in general, so that customers can visit and follow up on current problems AND progress.



3. Setting up a minimal Auto Voice Response system on our PBX such as (Press 1 for current Network Status, 2 to speak to a representative), so customers do not have to wait in line to get info.



K.

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
25/03/2012 22:06

@James, I can also confirm that your account was extended for 15 months on your account, however, in your case, as you have paid to the agent, you need to ask for an invoice to them as we will invoice the agent for your account and cannot provide you with an invoice directly as it will cause duplication in our accounting.



If you do not need an official invoice but a simple statement stating your expiration date, you can of course visit our office in Girne and the staff there will provide you with the document.



Kemal

yrys88


Joined: 08/08/2009
Posts: 1140

Message Posted:
26/03/2012 00:56

Hi James



If you need the expiration date pls send us an email at techmar.online@gmail.com and we will send it to you promptly.



Thanks

Yen

http://www.techmar-online.com

techmar.online@gmail.com

0392 8155441

keithcaley



Joined: 13/06/2008
Posts: 2521

Message Posted:
26/03/2012 08:16

Kemal,

I appreciate your detailed answer, and I'm delighted that you are even considering whether it is feasible or not.

If it turns out to be 'not', fair enough.

Keith.

juliamoons



Joined: 14/05/2009
Posts: 849

Message Posted:
26/03/2012 09:00

Kemal thanks for that explanation. I was confused when Skype came back at 10:55 but I could not access the internet. Nor could I ping google etc., I thought it was a problem with the DNS servers. It took until 11:50 until I was able to access the internet. I was wondering if it would be possible to use different DNS servers in the event that this may happen again.

erolz


Joined: 17/11/2008
Posts: 3456

Message Posted:
26/03/2012 10:10

juliamoons you can use alternative DNS servers should you wish too, however changing these settings is not something I would recommend for non technical users. Mistakes may render the internet connection unusable. For those that do wish to use alternate dns servers you can set them in your connections TCP/IPv4 settings. Google run public DNS servers on 8.8.8.8 and 8.8.4.4

Pippie


Joined: 02/12/2009
Posts: 1288

Message Posted:
26/03/2012 10:25

Hi Kemal,

Sorry to jump in on a different subject but didn't want to start a new thread when you're already watching this one!!



I want to sign up with Multimax when my current contract expires with Extend. Where is the Girne office? I'll mither them with the rest of my tedious questions.



Thanks.

erolz


Joined: 17/11/2008
Posts: 3456

Message Posted:
26/03/2012 10:28

Pippie the Multimax Girne office is in Karakum near the Lemar there, opposite the Kib-et steakhouse restuarant. The office is on the 2nd floor above a sports betting shop and the stairs up to it are behind the building.

keithcaley



Joined: 13/06/2008
Posts: 2521

Message Posted:
26/03/2012 11:07

erolz,

without 'jumping the gun' -'cos you're probably already thinking about this - am I right in thinking that using a different DNS servers wouldn't actually do Julia (msg 40) any good in the event that we lose contact with the outside world? - That is, she (& we!) wouldn't be able to reach those servers either!

I presume that you're trying to find a way to substitute some sort 'fake' entry into the cached DNS in the event that the cached entries 'expire', so that it will resolve all requests to the same address & point them to your Homepage or emergency announcement.

If I'm way off the mark, just tell me, and I'll shut up!

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
26/03/2012 12:22

@Keith, You are right and wrong because Julia is referring to a specific situation we have had yesterday(Sunday).



It is true that if the net is down in general, no DNS entries would work.



However, in the case of Internet being up and DNS servers being down, or somehow devices inability to pick up DNS servers automatically, entering manual/different DNS servers might provide (temporary in most cases until issues are resolved) solution.



Now, in the case Julia is describing, we had the following specific problem:

At around 10:50 yesterday(Sunday), the fiber link was back operational and we thought the internet is back up and running, it was, mostly.

When we made it back to the internet we have noticed there are still issues with customer, most of them telling us skype is not but they cant get through to websites.

(cont)

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
26/03/2012 12:27

After investigating the issue further, we have discovered that while the Internet was down, repeated DNS queries from thousands of customers has resulted in our 'smart' firewalls acknowledging this as some sort of attack on our system and blocked access for protection.



At this point in time, even though there was access to the Internet, using other DNS servers would NOT work as the firewall would block those requests as well.



Once we have identified the problem, we have amended the appropriate firewall rules and everything was back in normal operation, around 11:50am.



---

As erolz already stated, there must be a very specific reason why you would want to use DNS servers other than our own. We already have 2 DNS servers deployed, and also keep backups of them, so in case of a failure, it would take minutes, if not seconds to deploy a backup and have it up and running.



K.

greenbeans


Joined: 26/08/2011
Posts: 105

Message Posted:
26/03/2012 13:44

Excellent customer service. Thank you x

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
26/03/2012 14:47

Just to give all an update:

We have just moved to a new pair of fiberoptic cables for our Internet connection, all seems to be working well and the cause problems experienced over the weekend is now permanently resolved.



I again thank all our customers for their understanding.



Kemal

juliamoons



Joined: 14/05/2009
Posts: 849

Message Posted:
26/03/2012 16:04

Erolz and Kemal thank you both for those answers to my queries. I thought that the DNS servers were just taking too long to update. Now that you have reconfigured the fire walls I have no intention of changing the DNS servers. Went into Girne office today, am awaiting an email regarding upgrading to 10mbps.



Julia

keithcaley



Joined: 13/06/2008
Posts: 2521

Message Posted:
26/03/2012 16:28

Thanks for taking the time to enlighten us Kemal - nothing is ever simple is it?

timaloy


Joined: 30/06/2011
Posts: 171

Message Posted:
27/03/2012 01:30

to be honest kemal,theres been a few hiccups lately,all understandable,but the differance is,you as a buisnessman,have learnt very quickly on the importance of info,your response to your posts shows this,thats all people want ,a quick answer to probs,this you do okidokey,customers ALWAYS remember unanswered helplines etc etc

kbasat



Joined: 28/07/2011
Posts: 481

Message Posted:
27/03/2012 18:50

@timaloy, although there is room for improvement, I do not think we can ever be ready to answer ALL phone calls in a situation where there is a total internet downtime. I do not think it is in anyway feasible to employ 50 people with 50 phonelines just 'in case' we have this kind of problem again.



Customer may always remember unanswered helplines, but we are hoping that at the same time they would simply 'understand' when they later find out the extreme circumstances we were in at the time.



For all other times, we provide our customers with many ways to reach us. You can always email us at info@multimaxcyprus.com (preferred way for non-urgent matters), use our website http://www.mmcyp.com for online chat, call support line 0548 888 6629, or girne office 0548 870 0226.



Kemal